I haven't seen evidence to support the "load of crackpots" theory, though I suppose tying improving the art of rationality to FAI could have that effect. So might putting effort into highly implausible scenarios, though I personally see that as philosophy geeking rather than crackpottery.

I think it's the belief in the Singularity and cryonics which come off badly.