A single continuous line drawn left to right on a cream background: it begins as a dense tangled knot in brick red, loosens into wide open loops in terracotta, then straightens and folds back on itself into a stack of neat parallel lines in deep teal, like lines of text on a page.

Language Is the Greatest Compression Algorithm Humanity Ever Built

  • AI
  • language

Here is a claim I keep coming back to: natural language is a compression scheme, and an extraordinarily aggressive one. When I first wrote that sentence down I meant it as a metaphor. I have since come to think it is close to a literal description. Reality arrives at the human nervous system as a torrent, something on the order of ten million bits per second by one estimate from cognitive science.1 Speech and text leave us at roughly twelve bytes per second. Everything ever written down is an attempt to squeeze fragments of that torrent through a channel about a million times too narrow for it.

If that framing is right, it has a consequence that I think matters a great deal for AI: using language requires the thing that sits on the other end of any compression scheme. It requires a decompressor. For humans, that decompressor is a world model built from years of sensory experience. Large language models have to reconstruct one from text alone, and the more I sit with the numbers, the more remarkable it seems that this works at all. This essay is my attempt to work out the compression view carefully: where the numbers come from, what the history of writing looks like through this lens, and what it predicts about where language models will be strong and where they will be brittle.

How big the mismatch actually is

It's worth pausing on just how mismatched the two bandwidths are, because the whole argument rests on this.

Yann LeCun has laid out the input side repeatedly: each optic nerve carries roughly a million fibers at about ten bytes per second each, which puts vision alone at around 20 MB/s. A more conservative figure, from Wilson's 2000 review in Trends in Cognitive Sciences, puts total sensory throughput at about ten million bits per second, roughly 1.25 MB/s, once you account for processing upstream of awareness. Even the conservative number is enormous when integrated over time. A sixteen-hour waking day at 1.25 MB/s is about 72 GB, which sits comfortably alongside the 2009 UC San Diego estimate by Bohn and Short that the average American consumed about 34 GB of media per day, and that study counted only curated media. Run LeCun's visual estimate over a childhood, two million fibers at ten bytes per second across roughly sixteen thousand waking hours, and you find that a four-year-old has taken in on the order of a petabyte through the eyes alone.

Now the output side. An alphabet of about 22 letters carries roughly 4.5 bits per letter; an average English word of five characters is about 22 bits; speech at 150 words per minute works out to around 7 bytes per second, and LeCun's round figure for language, including encoding overhead, is about 12 bytes per second. Divide 20 MB/s by 12 bytes/s and you get a ratio of roughly 1.7 million to one. That is the compression ratio at which we routinely, casually, encode reality into words. My instinct when I first ran these numbers was that a channel that narrow should barely be able to carry anything of value. Civilization runs on it anyway, so the numbers must be missing something.

Writing got better by discovering what it could leave out

The history of writing reads, through this lens, as a sequence of increasingly bold decisions about what not to record.

The earliest recorded information was barely compressed at all. The cave paintings at Chauvet, some 36,000 years old, and Lascaux, around 17,000, are attempts to store visual experience as visual experience: a lossy copy of an aurochs, expensive to produce, bounded by what one artist can render. The first real leap was the realization that this fidelity was mostly wasted, because the audience shares your environment. You do not need to draw every detail of a deer for someone who has seen deer. You need only point at the concept. That insight, that shared experience lets you replace depiction with reference, is the founding move of symbolic writing, and it is a compression insight through and through.

From there it took three more steps, spread over two thousand years. Proto-cuneiform in Mesopotamia, around 3500 to 3000 BC, reduced whole scenes to stylized signs, a sheaf of grain for "grain," an ox head for "ox," about 705 signs in all, growing out of the clay-token accounting system Denise Schmandt-Besserat traced back to roughly 8000 BC. Then, in the early third millennium BC, scribes made the strange and decisive move of using signs for their sound rather than their meaning: the rebus principle, a picture of the sun standing for the syllable ud. This quietly changed what was being compressed, from the visual world to speech itself. And finally, around 1500 BC, the Proto-Sinaitic alphabet and its Phoenician descendant collapsed the entire space of human speech sounds into roughly 22 symbols, a system spare enough for ordinary traders to learn, and which radiated outward into the Greek, Latin, Arabic, and Hebrew scripts.

Each step discarded more of the surface and trusted more to the receiver.

Two megabytes of Tolkien against 150 gigabytes of film

Here is the comparison that convinced me the compression framing is more than a metaphor. The text of The Lord of the Rings, around 480,000 words, fits in roughly 2.4 MB.2 Peter Jackson's extended film trilogy runs about eleven and a half hours and occupies something like 150 to 190 GB on 4K Blu-ray. The text is about sixty thousand times smaller than the film of the same story.

If text were simply an impoverished version of film, a sixty-thousandfold reduction should leave almost nothing behind. But ask people who have experienced both, and you hear the same thing over and over: the books felt richer. The version of Rivendell that readers built in their heads outdid the one a major studio spent hundreds of millions of dollars rendering.

I think the resolution of this apparent paradox is the central fact about language. The text is a set of reconstruction instructions addressed to a mind that already contains sunsets, mountains, dusk, and valleys. When you read "The sun set behind the Misty Mountains, casting long shadows over Rivendell," your brain renders the scene: light, color, air, architecture, all pulled from a lifetime of stored sensory experience. My Rivendell, I suspect, borrows from the forests I wandered as a child. The reader is the decompression engine, and the richness lives on the reader's side.

This also tells you when language fails. What gets through is, roughly, the intersection of the author's world model and the reader's; what is lost is whatever the author encoded that the reader's model cannot reconstruct. A child reading that sentence renders a thinner scene than an adult, because the codebook doing the rendering is thinner. When I read a paper on tunneling magnetoresistance, I decompress it into barrier heights and spin-dependent densities of states; a friend outside physics reading the same abstract gets almost nothing, because the intersection is nearly empty. Seen this way, standardized education is a civilizational project to build compatible decompressors: curricula, SI units, mathematical notation, and disciplinary jargon all exist so that when one researcher compresses years of work into a few megabytes of journal article, another can faithfully expand it.

What this predicts about machines that learn from text alone

This is the part I actually care about. If understanding language means decompressing it, and decompressing it requires a world model, what happens when you try to build the world model from the compressed stream itself?

That is what a language model does, and the data asymmetry is stark. A large training corpus, on the order of 10^13 tokens, comes to about 20 TB, roughly all the quality public text on the internet. The four-year-old's visual intake alone was about a petabyte, fifty times more, before counting sound, touch, or proprioception. A human reading at 12 bytes per second would need well over a hundred thousand years to get through the LLM corpus, but of course no child learns language that way. The child's petabyte is what gives the twelve bytes per second something to point at. The model has only the pointers, and must infer the territory from the statistics of how pointers co-occur. It is a bit like trying to learn what music sounds like by reading a very large number of music reviews. The genuinely surprising empirical discovery of the last decade is how far that can get you; text at scale apparently constrains the world model far more than I would have guessed. Even so, the framework predicts, fairly specifically, where the gaps should remain.

The prediction is this: language models should be strongest where the text nearly is the territory, and weakest where the text is a savage compression of something irreducibly sensory. Mathematics and computer science sit at one extreme. Proofs, code, algorithms, and data structures are themselves text; the compression between the artifact and its description is mild, and this is, I suspect, a large part of why LLMs became competent programmers and proof assistants so quickly. Experimental physics, chemistry, and biology sit near the other extreme. The feel of a sample under cryogenic conditions, the hand knowledge of nanoscale devices, the eye that learns, after enough late nights at the instrument, which wiggles in a transport curve are physics and which are a loose cable: journal articles compress all of this brutally, on the assumption that the reader has stood where the author stood. A model that has never watched a fabrication run fail is missing precisely the codebook those papers assume.

Even the alignment phase fits the frame, though I hold this reading more loosely. Annotators correct the model where its output diverges from what a human world model would have produced, nudging the two into overlap one comparison at a time.

What would change my mind

Writing, then, has spent five thousand years perfecting a trade: representational richness surrendered for transmission efficiency, at ratios that dwarf anything in digital signal processing, and the trade only works because every reader arrives carrying petabytes of sensory experience to serve as the shared codebook.

For AI, this reframes the question worth asking: how does a machine acquire the thing text points at? My expectation, if this essay is right, is that scaling text keeps improving performance in mathematics and computer science, where the objects of study are themselves text, and that the stubborn residual failures will cluster in experimental physics, chemistry, and biology, whose literatures assume embodied experience. For materials science and experimental physics, that implication is concrete. A model meant to reason about materials the way an experimentalist does will need to train on experimental data itself: the raw measurement traces, the characterization images, the growth runs that failed, everything the papers compressed away on the assumption that the reader had stood at the instrument. I hold that view with moderate confidence, and it is testable: if reading papers alone produces reliable physical intuition of the kind experimentalists carry, the compression framework as I've stated it is wrong somewhere, and finding where would itself be interesting. Until then, I think of today's models as brilliant readers of maps who have never been to the territory.

Footnotes

  1. Wilson (2000), Trends in Cognitive Sciences. The sensory-bandwidth figures in this essay are order-of-magnitude estimates, and the specific numbers (LeCun's 20 MB/s for vision, 12 bytes/s for language) are his public calculations mentioned in his talks. The argument needs only the ratio to be enormous, which every estimate agrees on. ↩

  2. Word counts for the trilogy excluding appendices range from about 455,000 to 481,000 depending on the edition and counting method; at roughly five bytes per word this is 2.3 to 2.4 MB, consistent with typical e-book file sizes of 1 to 2 MB. ↩