I think Derbyshire is partly right, partly silly, and would have a lot less reason to be nervous around black people if he learned "how to act", as those scary strange black folks say.
What do you mean by "how to act"? If you mean it's necessary to adopt a different set of behaviors when around blacks, this is precisely Derbyshire's point.
But Derbyshire doesn't really work as a general signal flag for racism. Racial essentialism is one obvious answer: the idea that races are essential categories like species.
What do you mean by "essentialism"? After all the distinction between species isn't always clear either.
The "colorblind" folks aren't always nasty (..) but they are wrong.
What specific statements of theirs do you believe to be wrong.
If you mean it's necessary to adopt a different set of behaviors when around blacks, this is precisely Derbyshire's point.
There's a difference between "change of behavior" and "RUN!" A party full of black people will tend to have a different atmosphere than a party full of white people, just like parties with different mixes of age groups and genders will have a different atmosphere. Normal, healthy people can pick up on social cues. Black people are used to white people being scared of them. They have a good chance of noticing, ...
Related: Heuristics for Evaluating the Soundness of the Academic Mainstream, Admitting to Bias, The Ideological Turing Test