Cobra Effect · Computing and information
The man who measured information
How Claude Shannon turned information into something that could be measured in bits.
7 cards, read aloud in 2:41, with a test and sources.
In 1937, a young student at MIT noticed that switches could do logic.
Claude Shannon was studying circuits built from relays, switches that open and close. He saw that the algebra of true and false could describe exactly how those circuits behave. His master’s thesis showed how to design and simplify switching circuits using that algebra. The same idea lies at the heart of every digital computer.
After the war, at Bell Labs, he asked what information really is.
Telephone lines, radio and telegraph all carried messages, and all suffered from noise. Shannon wanted a way to measure information itself, whatever a message was about. He set the meaning of messages aside, as irrelevant to the engineering problem.
In 1948, he published A Mathematical Theory of Communication.
The basic problem, he wrote, is reproducing at one point a message chosen at another. He measured information in binary digits, which he called bits. He credited the word bit to his colleague John Tukey.
A bit is the answer to one yes or no question.
Finding which of eight doors hides a prize takes three yes or no questions, but a coin toss takes only one. The more uncertain an outcome, the more information learning it gives you. Shannon called this measure of uncertainty entropy.
His most surprising result was about noise.
Every channel, from a phone line to a radio link, has a maximum rate called its capacity. Below that rate, he proved, messages can be coded so that errors become as rare as you like. Noise did not have to garble messages. It set a speed limit.
Engineers spent decades finding codes that come close to his limit.
His paper even described a simple error correcting code found by his colleague Richard Hamming. Mobile phones, the internet and signals from distant spacecraft all rely on the ideas he set out. Claude Shannon died in 2001.
So when something seems impossible to measure, ask what its simplest unit could be.
Information seemed far too vague to put a number on. Shannon found its smallest piece, the answer to a single yes or no. Once it could be measured, it could be engineered.
Sources
- A Mathematical Theory of Communication, Claude Shannon, 1948. The paper that founded information theory.
- A Symbolic Analysis of Relay and Switching Circuits, MIT DSpace. Shannon’s master’s thesis on logic in switching circuits.
- Shannon’s thesis on switching circuits, History of Information. A short history of the thesis and why it mattered.
Nearby ideas
- The question no machine can answer. How Alan Turing imagined the computer, and proved some questions have no answer.
- The codebreakers who read Enigma. How Polish mathematicians and Bletchley Park broke Germany's Enigma machine.
- The codes that fix their own mistakes. How a frustrated mathematician invented codes that find and fix their own errors.
- The tiny switch that replaced the valve. How a slab of germanium at Bell Labs gave rise to every modern chip.
- The notes that imagined a computer. How Ada Lovelace's notes on an unbuilt machine imagined what computers could do.