A long blog post explains why the author, a feminist, is not comfortable with the rationalist community despite thinking it is "super cool and interesting". It's directed specifically at Yvain, but it's probably general enough to be of some interest here.
http://apophemi.wordpress.com/2014/01/04/why-im-not-on-the-rationalist-masterlist/
I'm not sure if I can summarize this fairly but the main thrust seems to be that we are overly willing to entertain offensive/taboo/hurtful ideas and this drives off many types of people. Here's a quote:
In other words, prizing discourse without limitations (I tried to find a convenient analogy for said limitations and failed. Fenders? Safety belts?) will result in an environment in which people are more comfortable speaking the more social privilege they hold.
The author perceives a link between LW type open discourse and danger to minority groups. I'm not sure whether that's true or not. Take race. Many LWers are willing to entertain ideas about the existence and possible importance of average group differences in psychological traits. So, maybe LWers are racists. But they're racists who continually obsess over optimizing their philanthropic contributions to African charities. So, maybe not racists in a dangerous way?
An overly rosy view, perhaps, and I don't want to deny the reality of the blogger's experience. Clearly, the person is intelligent and attracted to some aspects of LW discourse while turned off by other aspects.
I don't think that gets at the core of the criticism.
I think the position is: "You shouldn't be allowed to argue that policy X is good in the abstract scenario A if policy is is dangerous in the world in which you are living B and the fact that you argue that X is good in A increases the chances that X will be adopted in B."
I'd suggest unpacking that "shouldn't be allowed".
To me, it reads something like:
"Let's say that in abstract scenario S, policy X sounds like a utility-maximizing proposal; but in the world we're living, policy X would hurt our neighbors A, B, and C. If we spend our social time chatting about policy X and how great it would be, and chide people who criticize policy X that they are not being good utility maximizers, we should predict that A, B, and C will see us as a threat to their well-being."
That last bit is the part I think a lot of this discussion is missing.