I am not sure that worrying about the perils of growth to the degree you suggest is wise. Given how difficult it is to separate personal dislikes from competence, it seems to me that having a process to identify and remove specific problems (X is scaring off the ladies, let's train him or boot him) is much better than trying to optimize the group (I have more fun when Y isn't there, let's stop inviting them).

I also suspect this isn't intended to be an ivory tower coterie, but a growing movement- which means you want all people above minimum competence regardless of their current skill level. If you've got that sort of growth atmosphere, you'll eventually get enough people that you can sort, and your immediate group will have more members of the average quality you want.