Cobra Effect · Thinking and evidence
Base rate neglect
A good test for a rare thing still gives mostly false alarms.
6 cards, read aloud in 2:21, with a test and sources.
Your test came back positive. The test is 99% accurate.
The disease is rare. About 1 in 1,000 people have it. You are sitting in the waiting room doing the sum in your head. Ninety nine percent. So it is almost certain. Isn’t it?
Try it with a thousand people instead of one.
Out of 1,000, one person has the disease. The test catches them. The other 999 do not have it. But the test is wrong 1% of the time, so about 10 of them test positive anyway. Eleven positives. One of them is sick.
So your positive result means about a one in eleven chance.
Nine percent, not 99. The test is fine. The disease is just so rare that the false alarms outnumber the real cases. Ignoring how rare the thing was to begin with is called base rate neglect. The base rate is the 1 in 1,000. It is the number nobody looks at.
1978. Harvard doctors get it wrong too.
Researchers put a version of this question to staff and students at Harvard Medical School. Fewer than one in five got it right. The commonest answer was 95%. Training in medicine had not helped. The big number felt like the answer.
It is everywhere a rare thing is being screened for.
Security checks at airports, where nearly everyone flagged is innocent. Fraud alerts on your card. The friend who is sure a stranger is a spy because he acts like one. Most people who act like spies are not spies. There are so few spies.
The fix is to count people, not percentages.
Whenever a test comes back positive, ask two questions. How common is this thing to begin with? And out of a thousand people like me, how many would test positive without having it? Gerd Gigerenzer showed that doctors get the sum right when it is put that way. So do you.
Sources
- Base rate fallacy, Wikipedia. The medical test worked through, the 1978 Harvard study, and the taxi cab version that Kahneman and Tversky used.
- Reckoning with Risk, Gerd Gigerenzer, 2002. How to turn percentages into counts of people so that doctors, lawyers and patients get the sum right. Published in the United States as Calculated Risks.
- Bayes’ theorem, Wikipedia. The formula underneath the story, with the drug test example worked in numbers. Skip the algebra if you like and read the examples.
Nearby ideas
- Survivorship bias. The failures you never see can reverse the lesson you draw.
- The birthday problem. Coincidences are far more likely than they feel.
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- Anchoring. The first number you hear bends every estimate that follows.
- The availability heuristic. Vivid stories make rare dangers feel common.
- The narrative fallacy. We invent causes for events because stories feel like explanations.
- Hindsight bias. Once you know the ending, it always looks obvious.
- Outcome bias. Judge a decision by what was known, not by how the dice fell.