You might have a point here, but it's not clear what the counterfactual is. As HBD advocates like to point out, many and perhaps most people (including minorities) behave in private as if they believed in HBD (for instance, by buying housing in "good" neighborhoods, choosing "good" schools and the like, where "good" is defined by demographics).
By this argument, the "less hateful voices" who are silenced include most of the population, even though these same people might want to enforce a ban on publicly accepting HBD, for social signaling reasons (i.e. not wanting to be perceived by others as a dogmatic racist loon). Interestingly, it's not clear to me how stable this equilibrium is.
many and perhaps most people (including minorities) behave in private as if they believed in HBD
Except those behaviors express belief in statistics, not that those statistics are biological in nature. Biological diversity does not have to be the cause for statistical trends. Have HBD adherents falsified other explanations for the statistics?
A long blog post explains why the author, a feminist, is not comfortable with the rationalist community despite thinking it is "super cool and interesting". It's directed specifically at Yvain, but it's probably general enough to be of some interest here.
http://apophemi.wordpress.com/2014/01/04/why-im-not-on-the-rationalist-masterlist/
I'm not sure if I can summarize this fairly but the main thrust seems to be that we are overly willing to entertain offensive/taboo/hurtful ideas and this drives off many types of people. Here's a quote:
The author perceives a link between LW type open discourse and danger to minority groups. I'm not sure whether that's true or not. Take race. Many LWers are willing to entertain ideas about the existence and possible importance of average group differences in psychological traits. So, maybe LWers are racists. But they're racists who continually obsess over optimizing their philanthropic contributions to African charities. So, maybe not racists in a dangerous way?
An overly rosy view, perhaps, and I don't want to deny the reality of the blogger's experience. Clearly, the person is intelligent and attracted to some aspects of LW discourse while turned off by other aspects.