Since Pearl asserts the first two, do I have to get rid of the idea that more qualifications lead to more pay? I can't see any other way out of the bind.
I believe so, with the caveat that this could be a reversal effect. That is, qualifications and pay may be positively correlated for the whole group because men have more of both than women, while for each subgroup the correlations are negative.
Consider the following situation:
Men have 60 points to spend at character creation. Each point can either be used on a year of schooling, or a dollar of salary, with a minimum of 10 in each.
Women have 30 points to spend at character creation. Each point can either be used on a year of schooling, or a dollar of salary, with a minimum of 10 in each.
Now Bob says, "Look! If we look at groups determined by salary, each man is more qualified than women in his cohort, by thirty years of schooling." Barbara says, "Look! If we look at groups determined by schooling, each woman earns less than men in her cohort, by thirty dollars."
If most people choose to spend their points equally, then the population will be dominated by the points (15,15) and (30,30), and so the Association of Higher Education will say "Look! Schooling and salary are positively correlated."
The causal diagram in this situation is clear, though: it's being male that leads to more points while the direct effect of schooling on salary is negative because those two come from the same pool of points.
That's a great explanation, thanks for writing it! From now on, I will use your explanation instead of mine.
An article by Judea Pearl, available here. It's quick at 8 pages, and worth reading if you enjoy statistics (though I think people who already are familiar with the math of causality1 will get more out of it than others2). I'll talk here about the part that I think is generally interesting:
I've never really liked the name "paradox," because what it seems to mean is "unintuitive phenomenon." (Wikipedia puts it as "something which seems false and yet might be true.") The trouble is that "unintuitive" is a two-place word, and it makes sense to think like reality, so that true things seem true to you, instead of still seeming false. (For example, when I first learned about Zeno's Paradox, I already knew calculus, and so Zeno's position was the one that seemed confusing and false.)
What I like most about Pearl's article is that it explicitly recognizes the importance of fully dissolving the paradox,3 and seems to do so. Simpson's Paradox isn't an unsolvable problem in statistics, it's a straightforward reversal effect--only if you use the language of causality.
1. My review of Causality gives a taste of what it would look like to be familiar with the math, but you'd need to actually read the book to pick it up. The Highly Advanced Epistemology 101 for Beginners sequence is relevant, and contains Eliezer's attempt to explain the basics of causality in Causal Diagrams and Causal Models.
2. Pearl discusses how you would go about using simulations to show that do calculus gives you the right result, but leaves it as an exercise for the reader.
3. How An Algorithm Feels From Inside is probably a better place to start than Dissolving the Question, and I can't help but echo a question from it: "So what kind of math design corresponds to [Simpson's Paradox]?"
See also: bentarm's explanation of Simpson's Paradox.