High-status serious people want to associate with high-status others. Allowing crackpots on the same venue dramatically decreases its attractiveness for quality participants.
You can't really give an elaborate justification for why in this particular case it's OK, because signals are shallow.
I declare this Open Thread open for discussion of Less Wrong topics that have not appeared in recent posts.