By analogy: Private property also includes (must include, in my opinion) the freedom to "impose" it on those who don't want it - If Alice has a bicycle which she considers to be her private property and Bob tries to take the bicycle because Bob doesn't believe in private property and doesn't respect the notion of "Alice's bicycle" in the first place, I'm damn well going to side with Alice in telling Bob to go away, and if necessary, threatening violence against Bob.
If you try to form a concept of "strictly voluntary private property" which only applies to those who want it, you hardly have private property at all - you have a standing invitation for people who disagree with you to take your stuff.
Back to the previous topic, the "strictly voluntary segregation" one has in America and most of the West is that if a hundred white people move to the middle of nowhere to establish a segregated whites-only village and build it from the ground up, they're only allowed to have that segregated village as long as every black person in America refrains from moving there. As Eugine notes, it's illegal for one race to take legal measures to keep the other race out. To generalize, members of one race need functionally unanimous, ongoing, unilaterally revocable permission from all members of the other race in order to be segregated at present. (I gather some ghettos, trailer parks etc. have this by being so unappealing that nobody wants to move there.)
The interpretation of "strictly voluntary" as requiring the agreement of the exact people one wants to avoid in the first place strikes me as a questionably high bar, similar to calling a lifetime imprisonment "strictly voluntary" on the grounds that you can leave as long as the warden gives you permission.
Would you say I'm not free to agree with my friend to met at the pub because if he doesn't show up the police won't do anything about it? Would you say my girlfriend and I aren't free to have a monogamous relationship because if I cheated on her the police wouldn't stop me and vice versa? Hell, would you say I'm not free to be on a diet because the police won't stop me if I overeat?
A long blog post explains why the author, a feminist, is not comfortable with the rationalist community despite thinking it is "super cool and interesting". It's directed specifically at Yvain, but it's probably general enough to be of some interest here.
http://apophemi.wordpress.com/2014/01/04/why-im-not-on-the-rationalist-masterlist/
I'm not sure if I can summarize this fairly but the main thrust seems to be that we are overly willing to entertain offensive/taboo/hurtful ideas and this drives off many types of people. Here's a quote:
The author perceives a link between LW type open discourse and danger to minority groups. I'm not sure whether that's true or not. Take race. Many LWers are willing to entertain ideas about the existence and possible importance of average group differences in psychological traits. So, maybe LWers are racists. But they're racists who continually obsess over optimizing their philanthropic contributions to African charities. So, maybe not racists in a dangerous way?
An overly rosy view, perhaps, and I don't want to deny the reality of the blogger's experience. Clearly, the person is intelligent and attracted to some aspects of LW discourse while turned off by other aspects.