Cobra Effect · Thinking and evidence
Overconfidence
Why we feel more certain than our knowledge allows.
6 cards, read aloud in 2:47, with a test and sources.
Ten questions. Give a range you are 90% sure contains the answer.
How long is the Nile? How many countries are there? What year was the telephone patented? You do not need to know. You only need to be honest about how much you don’t. Make each range as wide as you like. Nine of your ten ranges should contain the right answer.
Most people get about half.
Marc Alpert and Howard Raiffa ran a version of this on Harvard business students in 1969. The true answers fell outside the students’ ranges far more often than they should have. The test has been repeated for fifty years on managers, doctors, engineers and students. Ranges meant to catch the answer nine times in ten catch it about five.
That is overconfidence. Not too proud. Too narrow.
It is not thinking you are great. It is thinking you know more precisely than you do. Nobody asked you to guess the Nile to the mile. They asked for a range, and you still made it too tight. Don Moore and Paul Healy split it three ways in 2008. Overestimating yourself, overranking yourself against others, and this one, overprecision. Overprecision is the stubborn one.
Experts are not immune. Some are worse.
Philip Tetlock spent twenty years collecting forecasts from political experts. On the questions he tracked, the experts as a group barely beat chance, and the most famous did worst. The people who did well kept score, hedged, and changed their minds in small steps.
Calibration is a skill. Weather forecasters have it.
When a forecaster says 70% chance of rain, it rains on about 70% of those days. Not because forecasters are wise. Because they are told, every single day, whether they were right. Fast, honest feedback trains the feeling of certainty to match the facts. Most of us never get the feedback. So the feeling never learns.
Keep score on yourself.
When you say you are sure, write down how sure. Seventy percent. Ninety. Then check. If your nineties come true half the time, you now know what your ninety is worth, and so does everyone who works with you. Widen the range until it is honest. Being right half the time is fine. Claiming ninety is the problem.
Sources
- Overconfidence effect, Wikipedia. The 90% interval test and its hit rates, the three kinds from Moore and Healy, and the work of Alpert and Raiffa. Search their title A Progress Report on the Training of Probability Assessors.
- Superforecasting, Philip Tetlock and Dan Gardner, 2015. Twenty years of scoring expert predictions, then a tournament to find the people who beat them. Calibration, keeping score and changing your mind in small steps.
- Calibrated probability assessment, Wikipedia. How calibration is measured and trained, with the interval test you can try on yourself.
Nearby ideas
- The Dunning Kruger effect. Why the least skilled are often the worst judges of their skill.
- The planning fallacy. Why projects nearly always take longer than we plan.
- Hindsight bias. Once you know the ending, it always looks obvious.
- The halo effect. One good quality colours how we judge all the others.
- Motivated reasoning. We set a lower bar for evidence we want to believe.
- The Texas sharpshooter fallacy. Draw the target after the shots and any pattern looks meaningful.
- The Semmelweis reflex. Why experts reject evidence that says they have done harm.
- The illusion of explanatory depth. We think we know how things work until we try to explain them.