If you could detect that easily, if everyone, just looking at you, could say "oh he's probably dumb" or "oh he's probably smart", it would greatly amplify the base unfairness of being slightly more or less intelligent. If skin color really has a strong correlation with intelligence, it would be somehow like tatooing people their IQ score on their forefront.
You seem to have the unstated assumption that fairness is desirable. What if the people who control policy WANT to perpetuate unfairness which happens to benefit them? What if perpetuating that unfairness would happen to benefit you? Why would you want to give up an advantage?
Then they'd have an incentive to perpetuate a belief that their race is smarter on average.
But people tend to look negatively on open attempts to create an uneven playing field. "If this person will try to change the playing field so that they personally come out ahead at others' expense, would they do the same at my expense?" Unless you know that you and this other person mutually see each other as ingroup members, and the groups they're disadvantaging as outgroup members, you'd have reason to suspect that they'd act against your social interests.
Idang Alibi of Abuja, Nigeria writes on the James Watson affair:
An intriguing opening. Is Idang Alibi about to take a position on the real heart of the uproar?
Darn, it's just a lecture on personal and national responsibility. Of course, for African nationals, taking responsibility for their country's problems is the most productive attitude regardless. But it doesn't engage with the controversies that got Watson fired.
Later in the article came this:
This intrigued me for two reasons: First, I'm always on the lookout for yet another case of theology making a falsifiable experimental prediction. And second, the prediction follows obviously if God is just, but what does skin colour have to do with it at all?
A great deal has already been said about the Watson affair, and I suspect that in most respects I have little to contribute that has not been said before.
But why is it that the rest of the world seems to think that individual genetic differences are okay, whereas racial genetic differences in intelligence are not? Am I the only one who's every bit as horrified by the proposition that there's any way whatsoever to be screwed before you even start, whether it's genes or lead-based paint or Down's Syndrome? What difference does skin colour make? At all?
This is only half a rhetorical question. Race adds extra controversy to anything; in that sense, it's obvious what difference skin colour makes politically. However, just because this attitude is common, should not cause us to overlook its insanity. Some kind of different psychological processing is taking place around individually-unfair intelligence distributions, and group-unfair intelligence distributions.
So, in defiance of this psychological difference, and in defiance of politics, let me point out that a group injustice has no existence apart from injustice to individuals. It's individuals who have brains to experience suffering. It's individuals who deserve, and often don't get, a fair chance at life. If God has not given intelligence in equal measure to all his children, God stands convicted of a crime against humanity, period. Skin colour has nothing to do with it, nothing at all.
And I don't think there's any serious scholar of intelligence who disputes that God has been definitively shown to be most terribly unfair. Never mind the airtight case that intelligence has a hereditary genetic component among individuals; if you think that being born with Down's Syndrome doesn't impact life outcomes, then you are on crack. What about lead-based paint? Does it not count, because parents theoretically could have prevented it but didn't? In the beginning no one knew that it was damaging. How is it just for such a tiny mistake to have such huge, irrevocable consequences? And regardless, would not a just God damn us for only our own choices? Kids don't choose to live in apartments with lead-based paint.
So much for God being "just", unless you count the people whom God has just screwed over. Maybe that's part of the fuel in the burning controversy - that people do realize, on some level, the implications for religion. They can rationalize away the implications of a child born with no legs, but not a child born with no possibility of ever understanding calculus. But then this doesn't help explain the original observation, which is that people, for some odd reason, think that adding race makes it worse somehow.
And why is my own perspective, apparently, unusual? Perhaps because I also think that intelligence deficits will be fixable given sufficiently advanced technology, biotech or nanotech. When truly huge horrors are believed unfixable, the mind's eye tends to just skip over the hideous unfairness - for much the same reason you don't deliberately rest your hand on a hot stoveburner; it hurts.