Cobra Effect · Statistics and probability
Simpson’s paradox
A trend in every group can reverse when the groups are combined.
6 cards, read aloud in 2:43, with a test and sources.
Berkeley, 1973. The university is accused of turning women away.
The numbers look damning. That autumn 8,442 men applied to graduate school and 44% got in. 4,321 women applied and 35% got in. The gap is far too big to be chance. Somebody is going to be sued.
Then they look department by department.
Three researchers went through the admissions one department at a time. In most of them, women were admitted at the same rate as men or slightly higher. A few departments tilted each way, but added up, the tilt ran slightly in favour of women. The overall gap was real. The unfairness it seemed to prove was not there.
Women had applied to the hard departments.
Men applied in large numbers to engineering and the sciences, which took most applicants. Women applied in large numbers to English and the humanities, which took few. Add the departments together and the crowded ones drag the women’s average down. The 1975 paper did note that the question of why women chose those departments was a fair one to ask elsewhere.
That is Simpson’s paradox.
A trend that appears in every group can reverse when the groups are combined. Edward Simpson described it in 1951, though Karl Pearson and Udny Yule had seen it decades earlier. Colin Blyth gave it Simpson’s name in 1972. It is not a trick of the numbers. It is a trick of the mixing.
It can kill people if you read it the wrong way.
A 1986 study compared two ways of removing kidney stones. For small stones, open surgery did better. For large stones, open surgery did better. Overall, the other treatment did better. Because the surgeons had sent the hard cases to surgery, and the easy ones to the other treatment.
When the total says one thing and the parts say another, ask what is being mixed.
Find the hidden variable. Department. Stone size. Severity. Then decide which level answers your question. If you are choosing a treatment for one patient, the part is the truth. If you are asking why the whole differs, the mixing is the story. Never trust a total until you have seen the pieces.
Sources
- Simpson’s paradox, Wikipedia. The Berkeley numbers, the kidney stone table with all the figures, the batting averages example, and the history from Pearson to Blyth.
- The Book of Why, Judea Pearl and Dana Mackenzie, 2018. Pearl’s argument that the paradox dissolves once you draw the causes as a diagram. A whole chapter on it, with several worked examples.
- Simpson’s Paradox, Stanford Encyclopedia of Philosophy. The careful version. What exactly reverses, why it is not really a paradox, and what it means for deciding which level of the data to trust.
Nearby ideas
- Berkson’s paradox. Filtering who you look at can create links that are not there.
- Base rate neglect. A good test for a rare thing still gives mostly false alarms.
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- The law of small numbers. Small samples swing wildly, and the swings look like signals.
- The prosecutor’s fallacy. A rare match is not the same as a small chance of innocence.
- The St Petersburg paradox. A huge prize is worth less to you than its average suggests.
- Benford’s law. In real data, numbers starting with 1 far outnumber those starting with 9.
- The hot hand. Streaks are mostly chance, though not always entirely.