That's coz it's not your neighborhood, in the property ownership sense.

to later on contradict 'emselves a bit and complain about the lack of freedom to impose segregation on those who don't want it.

My point is that under US law segregation is illegal even for those who want it. Even if everybody in the neighborhood wants it to be segregated and signs a contract to keep it that way, the contract is illegal under US law and members can be punished if they refused to sell to someone on the basis of race.