Cobra Effect · Science and philosophy
The replication crisis
Why many famous findings vanished when others tried them again.
7 cards, read aloud in 3:28, with a test and sources.
2011. A respected journal publishes proof that people can see the future.
Daryl Bem, a senior psychologist at Cornell, ran nine experiments and found that students could guess where an image would appear before the computer chose. The methods were standard. The statistics were standard. Which raised an awkward question. If standard methods could prove this, what else had they proved?
The same year, three researchers showed how easy it was.
Joseph Simmons, Leif Nelson and Uri Simonsohn showed that, with a few ordinary choices, they could prove that listening to a song made people younger. Stop collecting data when the result looks good. Try several measures and report the one that worked. Drop the odd participant. Each choice looked innocent. Together they could prove anything.
2015. A hundred famous findings, tested again.
The Open Science Collaboration, led by Brian Nosek, redid 100 published psychology studies from three top journals. Ninety seven of the originals had found a clear effect. Only 36 of the repeats did. Where an effect did survive, it was on average about half the size first reported.
That is the replication crisis.
A large share of published findings, in psychology and beyond, do not hold up when someone tries them again. Not fraud, mostly. Small samples, flexible analysis, and journals that only print results that worked. Medicine, economics and cancer biology have found the same rot. The textbooks were full of things that were true once, in one lab, with 40 students.
Two famous casualties.
Power poses, 2010. Stand like a superhero for two minutes and your hormones shift and you feel bolder. A 2015 test with a much larger sample found no hormone change, and in 2016 one of the original authors said she no longer believed it. Ego depletion, the idea that willpower runs down like a battery. In 2016, 23 labs tried the standard test together and found almost nothing. Both had hundreds of citations. Both are in books you may own.
What changed after.
Researchers now often register the plan before they collect the data, so the target can’t move. Some journals accept a study on its design, before the results exist, so that a null result still gets printed. Samples got bigger. Data got shared. Repeating someone else’s study became respectable work. The crisis was bad news about the past and good news about the method. The method caught itself.
What to do when you hear about a study.
Ask how many people were in it. Forty is a hint, not a fact. Ask whether anyone has repeated it. And be most suspicious of the finding that is surprising, tidy and quotable, because that is exactly the kind that got printed without being checked. One study is a rumour. A replication is news.
Sources
- Estimating the reproducibility of psychological science, Open Science Collaboration, 2015. The paper, in Science, volume 349. Search the title for the free version. The Wikipedia page linked here covers it and the wider crisis in other fields.
- Science Fictions, Stuart Ritchie, 2020. Fraud, bias, negligence and hype, one chapter each, by a psychologist who lived through the crisis. Angry, clear and fair to the people who fixed things.
- Power posing, Wikipedia. The 2010 study, the 2015 failure to replicate, and the statement by Dana Carney in 2016 that she no longer believed the effect. A whole crisis in one idea.
Nearby ideas
- The law of small numbers. Small samples swing wildly, and the swings look like signals.
- Falsifiability. A claim that nothing could disprove tells you nothing.
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- Cargo cult science. Science can copy every outward form and still miss the honesty.
- Fermi estimation. Break an impossible question into small guesses you can make.
- Absence of evidence. Finding nothing only counts if you would have found it.
- Clarke’s third law. Advanced technology can look like magic, and so can a good trick.
- Schrodinger’s cat. A thought experiment built to show where the quantum rules run out.