Cobra Effect · Statistics and probability
The garden of forking paths
Choosing the analysis after seeing the data makes flukes look real.
7 cards, read aloud in 2:27, with a test and sources.
A researcher runs one test, on one hypothesis, and reports one number.
No fishing. No hunting through the data until something worked. She had the idea before she collected anything. And her result may still be nonsense.
Because she made a hundred small decisions after seeing the data.
Two respondents looked odd, so she dropped them. Age split neatly at 40, so she used that. The pattern was clearer in women, so she reported women. Each choice was defensible. Each was made with the data in view.
Andrew Gelman and Eric Loken gave this a name in 2013.
They called it the garden of forking paths, after a short story by Jorge Luis Borges. You do not need to walk every path to be lost. You only need the path you took to depend on what you saw.
This is why a result can be honest and still be junk.
A significance test asks how often chance alone would hand you something this strong. That question has an answer only if the test was fixed in advance. If you chose the test after the data, chance had far more ways to oblige you. The published number understates that.
Nobody here is cheating.
Gelman and Loken were explicit about that. The researcher ran one analysis and believes it was the only one she could have run. The other analyses exist anyway, in the branches she would have taken if the data had come out differently. That is enough to break the maths.
The fix is boring and it works.
Write the analysis down before you see the data, and lodge it somewhere you cannot edit. If you cannot do that, report every path you considered, not just the one you walked. Or walk the whole garden and show the spread of answers it gives.
And when you are reading somebody else’s result.
Ask one question. Was this the analysis they planned, or the analysis that worked? If the split is oddly specific, if the finding lives in one subgroup, if the outcome is not the obvious one, you are looking at a path. Ask what else they could have found.
Sources
- The Statistical Crisis in Science, Andrew Gelman and Eric Loken, 2014. The published version of the 2013 argument, in American Scientist. The forking paths, with worked examples of honest researchers going wrong.
- The Garden of Forking Paths, Jorge Luis Borges, 1941. The short story the name comes from. A novel that is also a maze, in which every choice is taken at once. Read it for the image, not the statistics.
- Data dredging, Wikipedia. The louder cousin: hunting through data until something comes up significant. Useful for the contrast, because forking paths does the same damage with nobody hunting.
Nearby ideas
- The Texas sharpshooter fallacy. Draw the target after the shots and any pattern looks meaningful.
- The replication crisis. Why many famous findings vanished when others tried them again.
- The law of small numbers. Small samples swing wildly, and the swings look like signals.
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- The Monty Hall problem. Why switching doors wins twice as often as staying put.
- The birthday problem. Coincidences are far more likely than they feel.
- The gambler’s fallacy. Chance has no memory, so nothing is ever due.
- Berkson’s paradox. Filtering who you look at can create links that are not there.