Cobra Effect · Statistics and probability
The Literary Digest poll
A small sample chosen well beats a huge one of the wrong people.
7 cards, read aloud in 2:14, with a test and sources.
1936. A magazine posts out 10 million ballot cards.
The Literary Digest had picked every presidential winner since 1916. This time it went bigger than anyone had ever gone. About 2.4 million people filled in a card and sent it back.
The verdict was confident. Alf Landon would win.
The magazine gave him 57% of the popular vote. A sample of millions felt beyond argument. Then the votes were counted.
Franklin Roosevelt won 46 of the 48 states.
He lost only Maine and Vermont. The Digest had not been out by a little. It had been out by a landslide. The magazine folded within 18 months.
The problem was not the size. It was the list.
The names came from telephone directories, car registrations and the magazine’s own subscribers. In 1936, in the depths of the Depression, a car and a telephone meant money. The poll asked millions of people. It asked the wrong millions.
And the people who bothered to reply were not a fair half either.
Later study of the failure argued that who answered mattered even more than who was asked. People with a grievance post the card back. The contented leave it on the hall table.
A young pollster with 50,000 names got it right.
George Gallup asked a far smaller group, chosen to match the country rather than a mailing list. He called the election for Roosevelt and made his name overnight. A small fair sample beats a huge lopsided one. Every time.
So the question is never how many. It is who is missing.
Before you trust a survey, ask how the names were found. Ask who could not answer, and who chose not to. A million voices from one side of town is still one side of town.
Sources
- The Literary Digest, Wikipedia. The magazine, the 1936 poll and the numbers. For the argument that refusing to reply mattered most, search for Peverill Squire, 1988, on why the poll failed.
- How to Lie with Statistics, Darrell Huff, 1954. Still the best short book on numbers that mislead. The opening chapter is about samples with a built in lean, and it uses this poll.
- George Gallup, Wikipedia. The man who beat the magazine with a fraction of the replies, and who later got 1948 badly wrong himself.
Nearby ideas
- Survivorship bias. The failures you never see can reverse the lesson you draw.
- The law of small numbers. Small samples swing wildly, and the swings look like signals.
- Base rate neglect. A good test for a rare thing still gives mostly false alarms.
- The garden of forking paths. Choosing the analysis after seeing the data makes flukes look real.
- Regression to the mean. Extreme results drift back towards normal, whatever you do.
- The Monty Hall problem. Why switching doors wins twice as often as staying put.
- The birthday problem. Coincidences are far more likely than they feel.
- The gambler’s fallacy. Chance has no memory, so nothing is ever due.