For example: is it racist/sexist to point out the differences in average IQ between the people of different races/genders?

It depends on what relevance it has, and on what is being left out. Someone once told me that GW Bush must be smarter than Obama because he is white. That's an intellectual fallacy even if it isn't boo-word racism.

In my experience these words, especially when spoken by the offended, frequently mean "you are making an argument/stating a potential truth that I don't like".

In my experience, references to "human biodiversity" are frequently presented as if they are value neutral, but frequently aren't because of the factors mentioned above.