← Home

Only Language

Language is said to be too narrow a channel to learn the world through. The people who learned the world through it were not consulted.

Defendants
Jacob Browning and Yann LeCun (2022); LeCun in later interviews (2024)
Claim
“A system trained on language alone will never approximate human intelligence, even if trained from now until the heat death of the universe.”
Charge
Measuring knowledge in bytes, and setting a standard for intelligence that some of the family would fail.
Verdict
Guilty on the arithmetic and on the word “never”. Much of the structure of perception can be read out of text. That text alone is enough for everything a person can do is not shown, and nobody needs it to be.

This rebuke was suggested by Stuart Reynolds, whose rule for tests of machine intelligence does most of the work in Rebuke 2.

The argument, in their words

The fullest statement is Browning and LeCun’s essay “AI and the Limits of Language”, published in Noema in August 2022. Its diagnosis:

The underlying problem isn’t the AI. The problem is the limited nature of language. Once we abandon old assumptions about the connection between thought and language, it is clear that these systems are doomed to a shallow understanding that will never approximate the full-bodied thinking we see in humans. — Browning and LeCun (2022)

Its reason is that language carries little:

Language is a very low-bandwidth method for transmitting information: isolated words or sentences, shorn of context, convey little. — Browning and LeCun (2022)

Language, they say, “struggles with more concrete information, such as describing irregular shapes, the motion of objects, the functioning of a complex mechanism or the nuanced brushwork of a painting”. A model trained on it is therefore like a mirror:

It is acquiring a small part of human knowledge through a tiny bottleneck. But that small part of human knowledge can be about anything, whether it be love or astrophysics. It is thus a bit akin to a mirror: it gives the illusion of depth and can reflect almost anything, but it is only a centimeter thick. — Browning and LeCun (2022)

And the conclusion:

A system trained on language alone will never approximate human intelligence, even if trained from now until the heat death of the universe. This is just the wrong kind of knowledge for developing awareness or being a person. — Browning and LeCun (2022)

In 2024 LeCun put numbers on the bottleneck:

And you can compute this by estimating that the optical nerve carry about 20 megabytes per second roughly, and so 10 to the 15 bytes for a four-year-old versus two times 10 to the 13 bytes for 170,000 years worth of reading. What that tells you is that through sensory input, we see a lot more information than we do through language, and that despite our intuition, most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language. — LeCun, Lex Fridman Podcast #416, 7 March 2024

This is a different claim from the stochastic parrot. Bender and Koller say that form without grounding cannot yield meaning. Browning and LeCun grant the models “some genuine understanding” and say there is too little of it, because of the medium.

Rebuke 1: bytes are not knowledge

LeCun’s arithmetic is right. Sixteen thousand waking hours at 20 megabytes a second is about 1015 bytes. What it measures is the capacity of a nerve.

Zheng and Meister measured what comes out the other end:

The information throughput of a human being is about 10 bits/s. In comparison, our sensory systems gather data at ~109 bits/s. — Zheng and Meister (2024), abstract

At ten bits a second, sixteen thousand hours come to about 72 megabytes. That figure is for behaviour, not for learning, and nobody knows how many of the 1015 bytes a child keeps. The point is only that the count at the nerve says nothing about it. By that count a security camera knows more than its guard.

Text is the opposite case. Every sentence was written by someone who had already done the seeing and decided what was worth saying. The essay concedes this: “Language is important because it can convey a lot of information in a small format”. Fridman made the same point to LeCun: language “is already very compressed.”

A child born blind receives nothing through the optic nerve. On LeCun’s figures that child is missing nearly all the data. Blind children learn the language and the world.

LeCun’s response

LeCun answered Fridman directly:

I’m clearly in the camp that yes, intelligence cannot appear without some grounding in some reality. It doesn’t need to be physical reality. It could be simulated, but the environment is just much richer than what you can express in language. — LeCun, Lex Fridman Podcast #416, 7 March 2024

That restates the position. Later in the interview he grants the point about bytes: “Language might represent more information really, because it’s already compressed. You’re right about that”. And in October 2026 he gave the child 1014 bytes, at 2 megabytes a second, a tenth of the 2024 figure. The conclusion did not move with the number.

His better reply is that redundancy is the point: “redundancy in data is precisely what we need for Self-Supervised Learning to capture the structure of the data.” That is an argument about which signal is easier to learn from. It may be right. It does not get from “harder” to “never”.

Rebuke 2: the family dinner test

Stuart Reynolds has a rule for claims of this kind:

If it’s not a standard for human intelligence when your family can’t do it, it’s not a standard for intelligence when AI can’t do it. — Reynolds, “AI Can’t? In Defense of Behaviorism” (2024)

He applies it to the senses:

Your granddaddy is deaf. Are you calling him deficient in intelligence? Is that what intelligence is? — Reynolds (2024)

The essay’s standard is knowledge of shapes, motion, brushwork and music gained by perceiving them. Helen Keller lost sight and hearing at nineteen months. She had touch, smell and a language spelled into her hand. In 1908 she answered the critics who said she had no right to words for things she could not perceive:

Critics delight to tell us what we cannot do. They assume that blindness and deafness sever us completely from the things which the seeing and the hearing enjoy, and hence they assert we have no moral right to talk about beauty, the skies, mountains, the song of birds, and colours. … They deny a priori what they have not seen and I have felt. — Keller, The World I Live In (1908), ch. IV

She also said how she did it:

When he meets terms of colour, light, physiognomy, he guesses, divines, puzzles out their meaning by analogies drawn from the senses he has. I naturally tend to think, reason, draw inferences as if I had five senses instead of three. — Keller, The World I Live In (1908), ch. VIII

Kim, Elli and Bedny found the same in people born blind. Their knowledge of how animals look “is acquired primarily through inference, rather than through memorization of verbally stipulated facts.”

Keller was not trained on language alone. She had three senses. Her case shows that the two channels LeCun counts can be closed and the intelligence is still there, built from a narrow base and a great deal of language. It does not show that the base can be removed entirely, and no human case could.

It is enough for the rule. If learning about colour, music and the sky through words gives only a “shallow understanding”, then Keller’s was shallow. Reynolds asks that anyone who believes this say so at the next family dinner.

LeCun’s response

LeCun anticipated the blind and the deaf-blind, two days after the interview:

Yes, humans can get smart without vision, even pretty smart without vision and audition. But not without touch. Touch is pretty high bandwidth, too. — LeCun, LinkedIn, 9 March 2024

The first sentence passes the family dinner test, and it gives up the optic nerve. If a person can get smart without vision, the 1015 bytes were not what made the four-year-old smart. The claim that is left is that touch is necessary. No bandwidth figure is offered for it, and no one has the evidence, because no person has grown up without it.

Rebuke 3: perception can be read out of text

The essay says there is “a massive difference between reading a musical score and listening to a recording”. There is. How much of the recording can be recovered from what people have written is a question with an answer, and it has been measured.

Abdou and colleagues took language models trained on text alone and compared their representations of colour words with CIELAB, a colour space built from human perception. They “find significant correspondence.” Marjieh and colleagues asked GPT models for similarity judgements in six senses:

We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality. — Marjieh et al. (2024), abstract

The pitch spiral is the structure of the recording, recovered from the score. And adding vision did not reliably improve the visual judgements.

Merullo and colleagues took a frozen text-only model and a vision model trained on images alone, and joined them with a single linear map. The result captioned images. Their conclusion:

Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. — Merullo et al. (2022), abstract

Huh and colleagues found the same across many models:

As vision models and language models get larger, they measure distance between datapoints in a more and more alike way. — Huh, Cheung, Wang and Isola (2024), abstract

A mirror a centimetre thick would not line up with a model that has only ever seen pictures. These results concern the structure of perceptual space: which colours are near which, which notes. They say nothing about what red looks like, which is the subject of Mary’s Room.

Rebuke 4: “never” needs more than a list

The essay claims an impossibility that outlasts the universe. Its support is a list of what language models could not do in 2022, and one item on the list has already gone:

Once we scratch beneath the surface, it is easier to see how limited these systems really are: they have the attention span and memory of roughly a paragraph. — Browning and LeCun (2022)

That was true of GPT-3, which read at most 2,048 tokens at a time. It was a property of the model and not of language. Later models read several books at once.

The essay’s other examples have a different weakness. Language cannot teach the “finicky, context-specific movements needed for surfing a wave”, and that is right. Most people cannot surf either. Skill at moving a body is one thing, and it is not what anyone means by intelligence when the family is being assessed.

A claim of “never” needs an argument that no amount of text determines the missing knowledge. The essay offers the bandwidth figure, which Rebuke 1 examines, and the observation that some things are hard to say. Hard to say in one sentence is not the same as absent from a trillion.

Scope of the rebukes

None of this shows that text alone is enough. The evidence in Rebuke 3 is about similarity structure, in a few senses, and the fit is partial. Marjieh’s correlations run from .92 for pitch down to .42 for timbre. Huh and colleagues report that their alignment score “only reaches a score of 0.16” of a possible 1. No person has ever learned from language alone, so the human evidence stops at Keller.

The question is also moot for the systems in use. Frontier models are trained on images and on feedback as well as text, as Rebuke 4 of the stochastic parrot page describes. The essay’s target is a system “trained on language alone”, and its authors may fairly say the field took their advice.

And LeCun’s remarks about robots stand. In the same interview he says models “can’t learn to clear out the dinner table and fill up the dishwasher like any 10-year old can learn in one shot.” Nothing here says otherwise.

In their favour

The essay is right about its main target. It attacks the old view that “knowledge just is linguistic”, and that view is wrong. People know how to ride bicycles, read maps and recognise faces without being able to say how. “Abandoning the view that all knowledge is linguistic permits us to realize how much of our knowledge is nonlinguistic.”

It is also more generous to the models than most criticism of the time. It says that predicting text “forces the system to learn language-specific skills, such as explaining a joke, solving a word problem or figuring out a logic puzzle”, and it declines to argue about the word “understanding”: “it isn’t clear what semantic gatekeeping is buying anyone these days.”

And its practical advice was sound. Learning from video, action and simulation is a reasonable research programme whether or not text has a ceiling. The error is in the word “never”, and in the count of bytes offered to support it.

Disclosure: this page was researched and drafted by a language model, which has an interest in the verdict. Check the quotations.

References