I think that the fact that there is a debate and that the "good guys" use name-calling instead of scientific arguments, increases also the number of people in the group A.
It's a bit like telling people not to think of an elephant, and then justify it by saying that elephant-haters are most obsessed about elephants, therefore thinking of an elephant is an evidence of being an evil person. Well, as soon as told everyone not to think of an elephant, this stopped being true.
Actually, it is more like not being allowed to talk about the elephant (...in the room. See what I did there?). Not talking about a subject is much easier than not thinking about it. And because everybody knows that talking about the elephant will cause you to be called an elephant hater and nothing good whatsoever will come of it in 95% of cases, the only people who continue to talk about elephants are people who care so strongly about the subject that they are willing to be called an elephant-hater just so that they can be heard. So that leaves people wh...
A long blog post explains why the author, a feminist, is not comfortable with the rationalist community despite thinking it is "super cool and interesting". It's directed specifically at Yvain, but it's probably general enough to be of some interest here.
http://apophemi.wordpress.com/2014/01/04/why-im-not-on-the-rationalist-masterlist/
I'm not sure if I can summarize this fairly but the main thrust seems to be that we are overly willing to entertain offensive/taboo/hurtful ideas and this drives off many types of people. Here's a quote:
The author perceives a link between LW type open discourse and danger to minority groups. I'm not sure whether that's true or not. Take race. Many LWers are willing to entertain ideas about the existence and possible importance of average group differences in psychological traits. So, maybe LWers are racists. But they're racists who continually obsess over optimizing their philanthropic contributions to African charities. So, maybe not racists in a dangerous way?
An overly rosy view, perhaps, and I don't want to deny the reality of the blogger's experience. Clearly, the person is intelligent and attracted to some aspects of LW discourse while turned off by other aspects.