If humans were perfect reasoners, your objection would be valid. But people are irrational, and will tend to over-discriminate based on race if minor discrimination (based on the amount of info race does provide) is socially allowed.

Here's a thought experiment: have a bunch of typical humans guess the intelligence of a group of subjects, knowing their test scores, grades, and job performance, but not race. Then have the same judges guess the intelligence of a second group of subjects, using all the above info plus race, and the fact that "on average, whites are smarter than blacks" without any quantitative data on by how much*. I consider it likely that the second group, despite having more information, will be less accurate than the first. Therefore, if race-based IQ differences exist, we should try to ignore them unless we know their magnitude and are confident in our own rationality.

• "On average, whites are smarter than blacks" is all most people will remember from an article on race and IQ, and they won't think to look up by how much.