Aargh. That's a good point and I clearly need to think about this more. I don't have a clear theory yet, but I'm going to brain-dump my thoughts on this topic.
Then how do you know what score you should get on the IAT? I don't know what an unbiased score would be, but an equal-for-both-groups score is most likely biased.
The score you should get on the IAT should be correlated to your conscious opinion. If you consciously think Palestinians are inferior, then you should be happy with an IAT score showing you think Palestinians are inferior. If you consciously think Palestinians are equal to Israelis, you should be trying to get an IAT score reflecting that equality. It's all about trying to get the unconscious mind to correspond to your rational beliefs.
I consciously assent to the proposition "Palestinians are more likely to be suicide bombers than some other groups, but it's still only a tiny fraction of their population" My unconscious probably believes something closer to "Palestinians = suicide bombers!" Further, my conscious mind stops way short of the proposition "All Palestinians are bad people." My unconscious mind probably believes this second proposition. I'd like my unconscious mind to get way closer to my conscious beliefs in both areas.
My conscious, rational brain believes that most Palestinians are probably decent people who have been driven to extremes by their situation. My conscious mind also believes that the best Middle East peace plan is one where everyone, Israeli or Palestinian, is considered equally deserving of happiness simply because they are human. That's my moral system, and yours may differ. The point is, that is my moral system, and I would like to be able to operate on it. If I'm going around subconsciously thinking that Palestinians are bad and don't deserve happiness, I can't enact my goals.
This ties into the halo effect and the horns effect, where people tend to classify others as either all good or all bad. My belief that Palestinians are sometimes suicide bombers probably makes me think that they're uglier, stupider, and meaner than they really are. A lot of that is mediated through that concept "bad", which is one reason I'm so interested in getting my link between "Palestinians" and "bad" down.
If you wanted to adjust moral value for the Israelis' greater economic value, you'd still need a way to make sure you're not over-adjusting, ie that your unconscious mind doesn't dislike the Palestinians even more than your conscious mind does. Alternately, you'd want to make sure you weren't going soft and that your unconscious mind liked the Palestinians more than your conscious mind thought they deserved. I can't think of an easy way to do that with the IAT, but I bet there's a complicated one if you imagine a hypothetical IAT with perfect reliability and let me get away with a mere proof-of-concept.
Hope that makes sense.
Continuation of: The Implicit Association Test
Response to: 3 Levels of Rationality Verification
I've not yet seen it pointed out before that we use "bias" to mean two different things.
Sometimes we use "bias" to mean a hard-coded cognitive process that results in faulty beliefs. Take as examples the in-group bias, the recall bias, the bad guy bias, and various other things discovered by Tversky and Kahneman.
Other times, we use "bias" to mean a specific faulty belief generated by such a process, especially one that itself results in other faulty beliefs. For example, Jews are sometimes accused of having a pro-Israel bias. By this we mean that they have a higher opinion of Israel than the evidence justifies; this is a specific belief created by the in-group bias. This belief may itself generate other faulty beliefs; for example, they may have a more negative opinion of Palestinians than the evidence justifies. It is both the effect of a bias, and the cause of other biases.
Let's be clear about this "more than the evidence justifies" bit. Hating Hitler doesn't mean you're biased against Hitler. Likewise, having a belief about a particular ethnic group doesn't mean you're biased for or against them. My Asian friends hate it when people sheepishly admit in a guilty whisper that they've heard Asians are good at academics. Asians are good at academics. Just say "55% chance an average Asian has a GPA above the American population mean" and leave it at that. This is one of Tetlock's critiques of the Implicit Association Test, and it's a good one. I'd probably link Asians to high achievement on an IAT, but it wouldn't be a bias or anything to get upset about.
And let's also be clear about this faulty belief thing. You don't have to believe something for it to be a belief; consider again the skeptic who flees the haunted house. She claims she doesn't belief in ghosts, and she's telling the truth one hundred percent. She's still going to be influenced by her belief in ghosts. She's not secretly supernaturalist any more than someone who gets "strongly biased" on the IAT is secretly racist. But she needs to know she's still going to run screaming from haunted houses, and IAT-takers should be aware they're still probably going to discriminate against black people in some tiny imperceptible way.
Okay, back to the example. So the President appoints Isaac, a synagogue-going Jew, as the new Middle East peace envoy. Due to some amazing breakthrough in the region, both the Israelis and Palestinians agree to accept whatever plan Isaac develops. Isaac's only job is to decide what long-term plan is best for both sides. And he's a good man: he has an honest desire to choose the maximum-utility solution.
Isaac legitimately worries that he has a bias for the Israelis and against the Palestinians. How can he test the hypothesis? He can take a hypothetical souped-up version of the Implicit Association Test1. He finds that yes, he has a strong pro-Israel anti-Palestine bias. Now what does he do?
He can try to route around the bias. This is the approach implicitly endorsed by Overcoming Bias and by rationalism in general. He can take the Outside View and look at successful approaches in other world conflicts. He can use some objective metric to calculate the utility of everything in Israel, and check to make sure neither group is getting an amount disproportionate to their numbers. He can open a prediction market on metrics of success, and implement whatever policies trades at the highest value. All of these will probably improve Isaac's solution a fair bit. But none of them are perfect. In the end, Isaac's the one who has to make a decision that will be underdetermined by all these clever methods, and Isaac is still biased against the Palestinians.
Or he can try to fight the bias.
Diversity workshops try to fight biases directly . These don't work, and that's no surprise. Diversity workshops are telling you, on a conscious level, that minorities really are great people, aren't they? Well, yes. On a conscious level, you already believe that. Isaac already knows, on a conscious level, that the Palestinians deserve a fair solution that protects their interests just as much as the Israelis do. A diversity workshop would be a flashy video in which a condescending narrator explains that point again and again.
We don't have a lot of literature on what does work here, but I predict a few things would help. Make some Palestinian friends, to build mental connections between Palestinians and positive feelings. Learn to distinguish between Palestinian faces. Read works of fiction with sympathetic Palestinian characters. I would say "live in Palestine" but by all accounts Palestine is a pretty grim place; he might do better to live in a Palestinian community in America for a while.
Those techniques aren't especially good, but I don't care. We know how to improve them. By making a group take the Implicit Association Test, applying a technique to them, giving them the test again, and seeing how their score changed, we gain the ability to test bias-fighting techniques. I wouldn't want to do this on one person, because the test only has moderate reliability at the individual level. But a group of a few dozen, all practicing the same technique, would be quite sufficient. If another group learns a different technique, we can compare their IAT score improvement and see which technique is better, or if different techniques are better in different circumstances.
Again, there's no reason why this method should be limited to racial biases. No matter how hard I try to evaluate policies on their merits rather than their politics, I am biased towards the US Democratic Party and I know it. This ought to be visible on an IAT, and there ought to be techniques to cure it. I don't know what they are, but I'd like to find them and start testing them.
What about the second method of overcoming bias, routing around it? The IAT is less directly valuable here, but it's not without a role.
In one of the IAT experiments, subjects evaluated essays written by black or white students. This is a fiendishly difficult task upon which to avoid bias. A sneaky researcher can deliberately select essays graded as superior by a blind observer and designate them "white essays", so anyone trying to take the easy way out by giving all essays the same grade can be caught immediately. I like this essay task. It's utterly open to any technique you want to use to reduce bias.
So give someone IATs until you find a group they're especially biased against - black people, Palestinians, Korean-Americans, frequentists; any will do. Then make them grade essays by the control group and the disliked group. Collect statistics correlating IAT bias with essay grading bias. If a person using a special technique to route around mental bias can grade essays more accurately than other people with the same level of IAT bias, that person has routed around their bias successfully.
So: How do we tell if a technique for routing around bias works? Test whether people are better able to conduct a rating task than their IAT scores would predict. How do we test a technique for fighting bias directly? See if it lowers IAT scores. All terribly inconvenient because of the IAT's low effect size and reliability, but with a large enough sample size or enough test-retest cycles the thing could be done. And the psychologists who transformed the Bona Fide Pipeline into the IAT may yet transform the IAT into something even more powerful.
This, then, is one solution to schools proliferating without evidence. With enough research, it could be turned into one of the missing techniques of rationality verification.
Footnotes
1: Remember, the IAT is only moderately good at evaluating individuals, and has a bad habit of changing its mind each time someone takes it. Much of what is in this essay would work poorly (though probably still better than nothing) with a simple IAT. But having someone take the IAT ten times over ten days and averaging the results might give a more accurate picture (I don't know of any studies on this). And in any case the IAT is quite good at comparing groups of people with sample size >1. And I expect that souped-up versions of the IAT will be out within a few years; these tests have gotten better and better as time goes on.