The best objection I can come up with is really more of a generalized diversity argument: that heterogenous genetic populations tend to be more robust against environmental stresses, and that the same is probably true for their memetic analogues. But there's nothing coding that objection to any particular phenotype, minority or otherwise, and I can certainly imagine scenarios where it might be outweighed by circumstances.

I don't think I'd personally care too much in the scenario presented -- it's conceptually squicky, but I wouldn't see it as a personal attack -- but then again I'm a white dude, so any of Yvain's caveats could just as well apply to me. My first impulse when I imagine painlessly homogenizing various points of contention where I do fall into the minority is to appeal to something similar to the above, though.