calling their opponents vaguely defined negative terms, like "horrible racists"

Curious. I say I have no position on the question, and immediately some words I use are exemplary of how bad the people on the other side from you are.

Anyway, for the avoidance of doubt, I do not call all "race realists" horrible racists. For instance, I think Lumifer and Jiro right here on LW are on record as believing that there are genuine racial IQ differences, and I do not think either of them is a horrible racist. (I don't know either of them well enough to be sure they aren't, but they haven't given me that impression so far.)

The people I call horrible racists are the ones who seize every available opportunity to rant about how awful black people are, how stupid people who aren't "race realists" are, etc.; whose negative comments about black people go well beyond anything that could be justified by halfway-plausible versions of "race realism"; who, in short, behave as I would expect someone to behave who seizes on the (alleged) scientific evidence with glee because it suits their pre-existing prejudices.

It is perfectly possible to believe that black people have lower average IQ than white people without being a horrible racist[1]. It is perfectly possible to believe that black people have higher average IQ than white people while being a horrible racist[2].

[1] Not necessarily without being a racist, depending on the definition of that contentious term.

[2] A position similar to this, but with a very different racial group in place of "black people", has been pretty common for a long time.