Cobra Effect · Statistics and probability
The law of small numbers
Small samples swing wildly, and the swings look like signals.
7 cards, read aloud in 3:02, with a test and sources.
A map of the United States, coloured by kidney cancer.
Daniel Kahneman tells it, from a study by Howard Wainer and Harris Zwerling. The counties with the lowest rates are mostly rural and sparsely populated. Clean air, clean water, fresh food. It makes sense. Now look at the counties with the highest rates.
They are also rural and sparsely populated.
Poor access to doctors, high fat diets, too much drink. That makes sense too. Both stories cannot be true. Neither is. The rural counties are at both ends of the list for one reason. They are small.
A tiny county can have a rate of zero, or ten times the average, by chance.
One case in a tiny county is an outbreak. None is a miracle. A city of a million never gets either. Its rate sits near the true average because it has so many people that the luck evens out. Small samples swing wildly. Large ones do not.
Amos Tversky and Daniel Kahneman named the mistake in 1971.
They called it belief in the law of small numbers. People expect a small sample to look like the whole population it came from. It doesn’t. It looks like the population plus a lot of noise. They found the belief in professional researchers, who planned studies too small to find what they were looking for.
The hospital question.
A big hospital delivers 45 babies a day, a small one 15. Over a year, which records more days when over 60% of the babies born are boys? Most people say about the same. It is the small hospital, roughly twice as often. Fifteen babies can easily run 60% boys. Forty five hardly ever do.
It is why the best schools are small, and so are the worst.
In the early 2000s a large foundation put money into small schools, partly because small schools were so common among the top performers. Kahneman tells the story. Small schools were just as common among the worst. The size was not the cause. It was the noise. The extremes of any list are where the small samples live.
The question that stops it.
How many? A restaurant with five reviews, a fund with three good years, a hunch from two customers. Before you believe an extreme result, ask how big the sample was. If it is small, the result may be telling you about the sample, not the world. Extreme and small usually go together, and the small is doing the work.
Sources
- Insensitivity to sample size, Wikipedia. The hospital question, the 1971 paper, and why people treat a sample of ten like a sample of ten thousand. Short.
- Thinking, Fast and Slow, Daniel Kahneman, 2011. Chapter ten is called The Law of Small Numbers. The kidney cancer map, the small schools, and Kahneman admitting that he and Tversky fell for it themselves.
- The Most Dangerous Equation, Howard Wainer, American Scientist, 2007. The kidney cancer counties and the small schools, from the statistician who found them, with the formula that explains why small groups sit at the extremes.
Nearby ideas
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- The gambler’s fallacy. Chance has no memory, so nothing is ever due.
- Base rate neglect. A good test for a rare thing still gives mostly false alarms.
- The prosecutor’s fallacy. A rare match is not the same as a small chance of innocence.
- The St Petersburg paradox. A huge prize is worth less to you than its average suggests.
- Benford’s law. In real data, numbers starting with 1 far outnumber those starting with 9.
- The hot hand. Streaks are mostly chance, though not always entirely.
- The law of large numbers. Averages settle down over many tries, but a run never gets corrected.