Cobra Effect · Thinking and evidence
Survivorship bias
The failures you never see can reverse the lesson you draw.
6 cards, read aloud in 1:39, with a test and sources.
1943. The bombers come home full of holes.
The Allied air force wants to add armour, but armour is heavy. So they map where the returning planes were hit. Wings, fuselage, tail. The obvious plan is to armour those.
A statistician named Abraham Wald says the opposite.
Armour the places with no holes. The engines. The cockpit. Because the planes hit there did not come home to be counted.
You were looking at the survivors.
The holes on the returning planes show where a bomber can be hit and still fly. The missing planes carry the missing data. That is survivorship bias. Judging by what made it through, and forgetting what didn’t.
It hides everywhere success is visible.
Every famous dropout who founded a company. Nobody interviews the dropouts who didn’t. The old buildings that look better made than modern ones. The badly made old buildings fell down.
The gym is full of people who kept going.
Ask them what works and you will hear what worked for people who kept going. The people the routine didn’t work for are not there to ask.
The question that breaks it.
Where are the ones that didn’t make it? If you cannot see them, you are looking at a sample that chose itself. Go and find the missing planes.
Sources
- A Method of Estimating Plane Vulnerability Based on Damage of Survivors, Abraham Wald, 1943. The wartime memoranda behind the bomber story. Search the title to find the reprint. The real work is statistics, and the armour the engines line is the summary that history kept.
- How Not to Be Wrong, Jordan Ellenberg, 2014. Opens with Wald and the bombers, then spends the rest of the book on the other ways numbers fool people. The best popular maths book of its decade.
- Survivorship bias, Wikipedia. The bombers, the mutual funds that quietly closed, the cats that fell from high floors, and the buildings that are still standing. A catalogue of missing planes.
Nearby ideas
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- Goodhart’s law. A measure turned into a target stops telling you the truth.
- The razors. Simple rules for cutting away explanations you do not need.
- The Dunning Kruger effect. Why the least skilled are often the worst judges of their skill.
- Confirmation bias. We look for evidence that agrees, and rarely test what could say no.
- The planning fallacy. Why projects nearly always take longer than we plan.
- Base rate neglect. A good test for a rare thing still gives mostly false alarms.
- Anchoring. The first number you hear bends every estimate that follows.