An article by Judea Pearl, available here. It's quick at 8 pages, and worth reading if you enjoy statistics (though I think people who already are familiar with the math of causality1 will get more out of it than others2). I'll talk here about the part that I think is generally interesting:
Any claim to a resolution of a paradox, especially one that has resisted a century of attempted resolution must meet certain criteria. First and foremost, the solution must explain why people consider the phenomenon surprising or unbelievable. Second, the solution must identify the class of scenarios in which the paradox may surface, and distinguish it from scenarios where it will surely not surface. Finally, in those scenarios where the paradox leads to indecision, we must identify the correct answer, explain the features of the scenario that lead to that choice, and prove mathematically that the answer chosen is indeed correct. The next three subsections will describe how these three requirements are met in the case of Simpson's paradox and, naturally, will proceed to convince readers that the paradox deserves the title "resolved."
I've never really liked the name "paradox," because what it seems to mean is "unintuitive phenomenon." (Wikipedia puts it as "something which seems false and yet might be true.") The trouble is that "unintuitive" is a two-place word, and it makes sense to think like reality, so that true things seem true to you, instead of still seeming false. (For example, when I first learned about Zeno's Paradox, I already knew calculus, and so Zeno's position was the one that seemed confusing and false.)
What I like most about Pearl's article is that it explicitly recognizes the importance of fully dissolving the paradox,3 and seems to do so. Simpson's Paradox isn't an unsolvable problem in statistics, it's a straightforward reversal effect--only if you use the language of causality.
2. Pearl discusses how you would go about using simulations to show that do calculus gives you the right result, but leaves it as an exercise for the reader.
3. How An Algorithm Feels From Inside is probably a better place to start than Dissolving the Question, and I can't help but echo a question from it: "So what kind of math design corresponds to [Simpson's Paradox]?"
I disagree with Yan's discussion there on two (closely related) points.
First, I don't think his claim B (that "more educated people get paid more") is correct, because note that it's not actually true for greens and blues separately. That is, it might be true that the total effect is that education leads to higher pay, but it's not true that the direct effect is that education leads to higher pay, which is the same as my model in this comment. It looks to me like the direct effect of education on income is neutral or negative but the indirect effect (through color) is positive. (I have some training in estimating correlations from looking at graphs, but actually computing it would be better.)
Second, that suggests to me that this is a garden-variety reversal effect (i.e. Simpson's Paradox), so I disagree with his claim that it differs in origin from Simpson's Paradox.
The core of this disagreement is what the conditions are on the noise. I think that the noise needs to be negatively correlated (in two dimensions, the major axis of the ellipse runs northwest-southwest) to allow this effect (which means it's just an reversal effect obscured by noise). If it's possible to get this effect with uncorrelated noise (in 2d, the noise is circular), or positively correlated noise (the major axis of the ellipse runs northeast-southwest), then I'm wrong and this is its own effect, but I don't see a way to do that.
[edit] Actually, now that I say this, I think I can construct a distribution that does this by hiding another reversal in the noise. Suppose you have a dense line of greens from (0,0) to (2,2), and a dense line of blues from (2,2) to (4,4). Then also add in a sparse scattering of both colors by picking a point on the line, generating a number from N(0,1), and then subtracting that from the x coordinate and adding it to the y coordinate. Each blue on the line will only be compared with greens that are off the line, which are either below or to the left of it. Each
An article by Judea Pearl, available here. It's quick at 8 pages, and worth reading if you enjoy statistics (though I think people who already are familiar with the math of causality1 will get more out of it than others2). I'll talk here about the part that I think is generally interesting:
I've never really liked the name "paradox," because what it seems to mean is "unintuitive phenomenon." (Wikipedia puts it as "something which seems false and yet might be true.") The trouble is that "unintuitive" is a two-place word, and it makes sense to think like reality, so that true things seem true to you, instead of still seeming false. (For example, when I first learned about Zeno's Paradox, I already knew calculus, and so Zeno's position was the one that seemed confusing and false.)
What I like most about Pearl's article is that it explicitly recognizes the importance of fully dissolving the paradox,3 and seems to do so. Simpson's Paradox isn't an unsolvable problem in statistics, it's a straightforward reversal effect--only if you use the language of causality.
1. My review of Causality gives a taste of what it would look like to be familiar with the math, but you'd need to actually read the book to pick it up. The Highly Advanced Epistemology 101 for Beginners sequence is relevant, and contains Eliezer's attempt to explain the basics of causality in Causal Diagrams and Causal Models.
2. Pearl discusses how you would go about using simulations to show that do calculus gives you the right result, but leaves it as an exercise for the reader.
3. How An Algorithm Feels From Inside is probably a better place to start than Dissolving the Question, and I can't help but echo a question from it: "So what kind of math design corresponds to [Simpson's Paradox]?"
See also: bentarm's explanation of Simpson's Paradox.