← Home

The Chinese Room

Of course the book doesn't understand Chinese. The book you're picturing couldn't order lunch.

Defendant
John Searle, “Minds, Brains, and Programs” (1980)
Claim
No program, however good, could understand anything merely by virtue of being run.
Charge
Passing off a ring binder as an object larger than the universe; begging the question from inside a locked room.
Verdict
Guilty. An intuition about a clerk, promoted to a theorem about minds.

The argument, in Searle’s words

From “Minds, Brains, and Programs” (Behavioral and Brain Sciences, 1980):

Suppose that I’m locked in a room and given a large batch of Chinese writing. Suppose furthermore (as is indeed the case) that I know no Chinese, either written or spoken … To me, Chinese writing is just so many meaningless squiggles. Now suppose further that after this first batch of Chinese writing I am given a second batch of Chinese script together with a set of rules for correlating the second batch with the first batch. The rules are in English, and I understand these rules as well as any other native speaker of English. — Searle (1980)

The rules get so good that the room’s answers pass for a native speaker’s:

From the external point of view — from the point of view of someone reading my “answers” — the answers to the Chinese questions and the English questions are equally good. But in the Chinese case, unlike the English case, I produce the answers by manipulating uninterpreted formal symbols. As far as the Chinese is concerned, I simply behave like a computer; I perform computational operations on formally specified elements. For the purposes of the Chinese, I am simply an instantiation of the computer program. — Searle (1980)

And the conclusion:

… it seems to me quite obvious in the example that I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing. — Searle (1980)

So running a program can’t be sufficient for understanding, and “strong AI” — the claim that an appropriately programmed computer literally has a mind — is false.

In 1990 he boiled it down to a tidy derivation:

  1. Programs are purely formal (syntactic).
  2. Minds have mental contents (semantics).
  3. Syntax by itself is neither constitutive of nor sufficient for semantics.

Therefore: programs are neither constitutive of nor sufficient for minds.

That's the whole thing. It has been reprinted, taught, and solemnly debated for going on half a century. I think it's a joke.

My instinctive rebuke: the “book” is a trick

Start with the prop. Searle could have described the mechanism in the room any way he liked, and he chose a book: a rulebook, some baskets of symbols, “bits of paper.” That choice is the whole argument.

Everybody knows what a book is. It sits on a desk. It is finite, inert and human-sized. It has never learned anything, wanted anything or got a joke. Ask whether a book plus a clerk could understand Chinese and the answer is obviously no — and the reader, having agreed to that, is marched off to a conclusion about every possible program.

Any intelligent apparatus whatsoever can, in principle, be unrolled into a book: tabulate what it would say in response to everything that could be said to it, and bind the pages. That works for a chatbot. It works equally for you — there is a book of everything you would say in every conversation you could have over the next hour. So the existence of a book version shows nothing about whether the original understands. But the unrolled book is not the object you pictured on the desk, and the gap is not a matter of degree.

Take chess, a vastly smaller problem than conversation. A book that plays chess by listing a reply to every way the game could have gone so far needs an entry for each line of play: Shannon's estimate is around 10120 of them. The observable universe contains about 1080 atoms. At one atom per entry, the chess book needs some 1040 universes' worth of paper.

Chess can cheat, because everything that matters about the past is captured by the current position, and a position-indexed book shrinks to a mere 1044 or so entries — roughly a planet.

Conversation can't cheat that way. There is no “board”; what a remark means depends on everything said before it. A book covering only the possible opening messages of a hundred characters, drawn from three thousand common ones, already has about 10347 entries — and a native speaker has to cope with the gibberish ones too. This is essentially Ned Block's lookup-table machine, and Block was admirably upfront that it could not exist.

The alternative is to roll the book back up: a rulebook of manageable length, plus scratch paper, plus a great deal of rule-following. That's a program, and now we have some idea of the scale. A modern language model with tens of billions of parameters needs on the order of 1011 arithmetic operations to emit one token. At one operation per second, without sleep, the man in the room produces a fragment of a word roughly every three thousand years, shepherding a working state of billions of numbers as he goes. That isn't a book in any sense that licenses intuitions about books either.

So there are two honest versions of the thought experiment. In one, the “book” is a static object bigger than a great many universes. In the other, it is a process running over geological time across a vast, intricately structured, constantly evolving state. Does that understand anything? Your intuition is — what, exactly? You don't have one. Nobody does.

Our intuitions about what books can do were formed on objects you can carry under one arm, and they are simply silent about objects forty orders of magnitude beyond the cosmos. Searle's version works only because the reader quietly substitutes the ring binder for the thing stipulated, consults their intuition about the ring binder — which is correct! ring binders don't understand Chinese! — and then applies it to something that is not a ring binder. Dennett coined the term intuition pump in his reply to this very paper, and this is the mechanism.

In fairness to Searle, in 1980 he had Roger Schank's story-understanding scripts in mind, and about those he was right: they didn't understand anything, and they were roughly ring-binder-sized. The sin was generalising from “this program doesn't” to “no program could” on the strength of a mental image.

Searle’s response

Block, Dennett and Pylyshyn all told him in 1980 that the argument runs on intuition. He answered:

When I now say that I at this moment do not understand Chinese, that claim does not merely record an intuition of mine, something I find myself inclined to say. It is a plain fact about me that I don’t understand Chinese. Furthermore, in a situation in which I am given a set of rules for manipulating uninterpreted Chinese symbols … it is still a fact about me that I do not understand Chinese. Indeed, it is the very same fact as before. — Searle (1980), Author’s Response

Nobody disputes that fact about him. The intuition doing the work is the next step, that therefore nothing else in the room understands either, and that one is an intuition about a ring binder. On the size of the binder he says nothing.

The luminous room. Paul and Patricia Churchland's parody deserves to be better known than the original. A man stands in a dark room waving a bar magnet up and down. Maxwell says oscillating electromagnetic fields are light. But look — the room stays pitch black! Therefore: “forces by themselves are neither constitutive of nor sufficient for luminance.” The man's intuition is impeccable and his physics is wrong, because he's off by about fifteen orders of magnitude in frequency and mistook the limits of his imagination for a fact about the world.

Searle’s answer (1990) was that the analogy fails because light and electromagnetism are both physical, whereas syntax has no physical, causal powers at all. Which is premise 3 again, and “causal powers” again; see below.

Rebuke 2: wrong level, wrong suspect

The argument asks one question — does the man understand Chinese? — and then answers a different one — does anything here understand Chinese? The man is one component of the system. Specifically, he's the dumbest component: the bit that fetches an instruction and executes it. He is the CPU. The claim under attack was never “the CPU understands”; it was that the running program does.

No individual neuron in your head understands English. Nor does any particular sodium channel, nor the hypothalamus taken alone. If “I checked one part and it didn't understand” were a valid inference, it would show that you don't understand this sentence either. An argument that proves brains can't think has proved too much.

This is the systems reply, and it was put to Searle before the paper was even published. His response is not an argument:

The idea is that while a person doesn't understand Chinese, somehow the conjunction of that person and bits of paper might understand Chinese. It is not easy for me to imagine how someone who was not in the grip of an ideology would find the idea at all plausible. — Searle (1980)

“Bits of paper” — there's the ring binder again, doing the work an argument should be doing. And “somehow the conjunction of eighty-six billion cells and some salty water might understand Chinese” sounds exactly as silly, and happens to be true of about a billion people.

Rebuke 3: “fine, I'll memorise the rulebook”

Searle’s substantive answer to the systems reply:

My response to the systems theory is quite simple: let the individual internalize all of these elements of the system. He memorizes the rules in the ledger and the data banks of Chinese symbols, and he does all the calculations in his head. The individual then incorporates the entire system. There isn’t anything at all to the system that he does not encompass. We can even get rid of the room and suppose he works outdoors. All the same, he understands nothing of the Chinese, and a fortiori neither does the system, because there isn’t anything in the system that isn’t in him. — Searle (1980)

But this just re-runs the same confusion with the furniture removed. If the man really has internalised a program capable of fluent Chinese conversation, there are now two things going on in one skull: Searle, who speaks English and is bored out of his mind, and a Chinese-speaking process he is implementing step by step. That Searle has no access to what the second process understands is not surprising — it is precisely what implementing something means. The x86 chip running a Python interpreter running a chess engine does not know the Sicilian Defence. Your laptop hosting a Windows virtual machine has not thereby learned where the Start menu is. Implementation is not identity, and introspective access doesn't flow downhill from the implemented to the implementer.

So the only thing the memorisation move establishes is: the implementer doesn't understand. Which everybody already granted. The question of whether the thing being implemented understands is left exactly where it was, and Searle answers it by consulting the implementer — the one party guaranteed not to know.

Searle’s response

Wilensky put the two-systems point to him: suppose someone asks the room, in Chinese, whether it understands Chinese, and it answers that of course it does. Searle:

Does that show, as Wilensky implies, that there is a subsystem in me that understands Chinese? As long as there is no semantic content attaching to these symbols, the fact remains that there is no understanding. — Searle (1980), Author’s Response

That is the conclusion, offered as the reason for the conclusion. Whether the symbols have semantic content for the subsystem is the entire question, and he settles it by asking the other occupant of the skull.

Rebuke 4: premise 3 is the conclusion wearing a hat

Look again at the derivation. Premises 1 and 2 are harmless. All the action is in premise 3: syntax is not sufficient for semantics. But “no amount of symbol-processing can amount to understanding” is just the denial of strong AI, restated in linguistics vocabulary. It is the very thing in dispute.

A functionalist thinks meaning comes from the right kind of causal and computational organisation; premise 3 says it doesn't. You can't refute someone by listing the negation of their view as an axiom.

Here it is in the 1980 paper, stated rather than argued:

Because the formal symbol manipulations by themselves don’t have any intentionality; they are quite meaningless; they aren’t even symbol manipulations, since the symbols don’t symbolize anything. In the linguistic jargon, they have only a syntax but no semantics. — Searle (1980)

The Chinese Room is supposed to be the evidence for premise 3. But we've just seen what the room delivers: a stage prop that's wrong by dozens of orders of magnitude (my instinctive rebuke) and a confusion of levels (Rebukes 2 and 3). So the formal argument rests on the premise, the premise rests on the thought experiment, and the thought experiment rests on the reader not thinking very hard about it. It's turtles, and the bottom turtle is a vibe.

Rebuke 5: “causal powers” to the rescue

If programs can’t produce understanding, what does? From the abstract:

“Could a machine think?” On the argument advanced here only a machine could think, and only very special kinds of machines, namely brains and machines with internal causal powers equivalent to those of brains. — Searle (1980)

And why simulation won’t do:

No one supposes that computer simulations of a five-alarm fire will burn the neighborhood down or that a computer simulation of a rainstorm will leave us all drenched. Why on earth would anyone suppose that a computer simulation of understanding actually understood anything? — Searle (1980)

Two problems.

First, the analogy is chosen to fit the conclusion. Some things are not reproduced by simulating them: rain, digestion, fire. Other things are: a simulated calculator really does add; a simulated chess player really does beat you; a simulated proof checker really does check proofs. The entire question is which kind of thing understanding is — more like wetness or more like arithmetic. Searle doesn't argue that it's like wetness. He just picks rain.

Second, which causal powers? In forty-odd years no list was ever supplied. Not carbon, he said; he allowed that Martians or machines made of other stuff might have the right powers. Not the neural organisation either — when offered a program that simulates the brain neuron by neuron, he swapped the binder for plumbing (and there’s that clerk again):

… imagine that instead of a monolingual man in a room shuffling symbols we have the man operate an elaborate set of water pipes with valves connecting them. … Each water connection corresponds to a synapse in the Chinese brain … But the man certainly doesn’t understand Chinese, and neither do the water pipes … The problem with the brain simulator is that it is simulating the wrong things about the brain. As long as it simulates only the formal structure of the sequence of neuron firings at the synapses, it won’t have simulated what matters about the brain, namely its causal properties, its ability to produce intentional states. — Searle (1980)

So the magic ingredient is not the material and not the organisation. It is defined only as “whatever brains have that programs lack.” That's not a theory of mind. It's an IOU for one, and it makes the position unfalsifiable: any system you build, however it behaves, can be declared to lack the powers, since nobody knows what they are.

A postscript from the 2020s

For forty years the room was safely hypothetical, so the stipulation “suppose the rulebook is good enough” cost nothing. We have now actually built rulebooks that hold up their end of a conversation in Chinese, and you can go and read the rules: they are matrices.

Whether those systems understand anything, and in what sense, is a live and interesting argument. But notice how the serious participants argue it: with evidence about generalisation, internal representations, grounding, and failure modes. Nobody on either side settles it by pointing at a GPU and observing that the GPU doesn't speak Chinese. That move was always empty. It just took the real thing turning up for it to look empty.

In Searle’s favour

Behaviour alone doesn’t logically entail understanding. Ned Block’s giant-lookup-table machine makes this point cleanly, and without the misdirection: what’s going on inside matters. But that is an argument for caring about internal organisation, which is the functionalist’s view, not Searle’s.

Where meaning comes from is a real question. How symbols come to be about the world, Harnad’s symbol grounding problem, is hard and open. The Chinese Room gestures at it. It doesn’t answer it, and “causal powers” certainly doesn’t.

And the 1980 programs really didn’t understand anything. As a complaint about overclaiming in early AI, fair enough. As a proof about all possible programs, no.

References