Well, there's a correlation between race and height, too, no doubt, but such correlation is utterly insignificant comparing to variance within either race - you don't say whites are taller or blacks are taller, there's very short black populations and very tall ones, and ditto for the whites whose variance is smaller.

The quantitative differences make a qualitative difference here.

Racists believe the correlation to be of greater significance than that of correlation between the height and intelligence. Based on fairly poor evidence - Raven's matrices are not this culture fair, they're culture fair in the sense that you can test British, Germans, French, Russian, and Chinese and Japanese with it, not in the sense that you can go and test some tribe that doesn't do much arithmetic nor is exposed to similar visual stimuli.

By the way given the diversity of blacks it would be utterly surprising if there is not a single ethnicity there with an average IQ greater than 100, as well as IQ with greater or smaller variance. (I would expect blacks to have larger variance than whites because they're plain more diverse, and mixed to have greater variance still)

Then the "rational" racists also object to use or even the existence of correlation between such racism and intelligence, conscientiousness, education, and other factors.