Cobra Effect · Statistics and probability
The prosecutor’s fallacy
A rare match is not the same as a small chance of innocence.
6 cards, read aloud in 2:37, with a test and sources.
1999. A mother is on trial for the deaths of her two baby sons.
Sally Clark is a solicitor in Cheshire. Her first son died at eleven weeks. Her second at eight weeks. No one saw anything. The medical evidence is disputed. Then a famous paediatrician gives the jury a number.
One in 73 million.
Sir Roy Meadow tells the court that the chance of a cot death in a family like hers is about 1 in 8,543. Two cot deaths, he says, is that number squared. One in 73 million. The jury convicts.
The number was wrong twice.
First, squaring assumes the two deaths are independent, like two coin tosses. They are not. Whatever raises the risk for one baby, genes, home, anything, raises it for the sibling. The Royal Statistical Society said so publicly in 2001. But the deeper error is what the number was taken to mean.
That is the prosecutor’s fallacy.
One in 73 million was the chance of two cot deaths, if she was innocent. It was heard as the chance that she was innocent. Those are different questions. To answer the second, you must ask how likely the other explanation is. And two murders by a mother are rarer than two cot deaths.
Rare is not the same as guilty.
The mathematician Ray Hill worked it through in 2002. Given two infant deaths in one family, two natural deaths were between 4.5 and 9 times more likely than two murders. Sally Clark’s conviction was overturned in 2003, after it emerged that medical evidence pointing to a natural cause had been withheld. She never recovered, and died in 2007.
Every rare match has this shape.
A DNA match found by searching a million records. A fraud pattern that only 1 in 10,000 honest people would show. The small number is the chance of the evidence if the person is innocent. Before you decide what it means, ask two things. How many innocent people would produce this evidence anyway? And how rare is the guilty explanation?
Sources
- Prosecutor’s fallacy, Wikipedia. The definition, the 1987 paper by Thompson and Schumann that named it, the Sally Clark case, and the Royal Statistical Society’s objection.
- The Drunkard’s Walk, Leonard Mlodinow, 2008. A tour of randomness and the ways people misread it, including how probability gets mangled in the courtroom.
- Sally Clark, Wikipedia. The whole case, from the two deaths to the 1 in 73 million figure, the appeals, the withheld evidence, and the other mothers whose convictions were reviewed afterwards.
Nearby ideas
- Base rate neglect. A good test for a rare thing still gives mostly false alarms.
- Bayesian updating. Change your mind in proportion to the strength of the evidence.
- The Texas sharpshooter fallacy. Draw the target after the shots and any pattern looks meaningful.
- The St Petersburg paradox. A huge prize is worth less to you than its average suggests.
- Benford’s law. In real data, numbers starting with 1 far outnumber those starting with 9.
- The hot hand. Streaks are mostly chance, though not always entirely.
- The law of large numbers. Averages settle down over many tries, but a run never gets corrected.
- Mean and median. Why the average can describe nobody at all.