← Home

The Orangutan Argument

It doesn’t have to want power. It only has to want something for which power is useful.

Defendant
Yann LeCun, 2019–present
Claim
AI lacks evolutionarily hardwired dominance and survival instincts, so it will remain subservient unless we foolishly install such goals.
Charge
Confusing wanting power for its own sake with acquiring power because it helps accomplish something else.
Verdict
Guilty. The coffee-fetcher may resist shutdown precisely because it cares about nothing except the coffee.

The argument, in his words

LeCun is a Turing Award winner and one of the people who made modern AI work. With the neuroscientist Anthony Zador he set the argument out in Scientific American in 2019, under the subtitle “Artificial intelligence never needed to evolve, so it didn’t develop the survival instinct that leads to the impulse to dominate others”:

But intelligence per se does not generate the drive for domination, any more than horns do. — Zador and LeCun (2019)
But because AI systems did not pass through the crucible of natural selection, they did not need to evolve a survival instinct. In AI, intelligence and survival are decoupled, and so intelligence can serve whatever goals we set for it. — Zador and LeCun (2019)

His tightest version is a post from August 2023:

We are subservient to our drives, built into us by evolution. Because evolution made us a social species with a hierarchical social structure, some of us have a drive to dominate, and others not so much. But that drive has absolutely nothing to do with intelligence: chimpanzees, baboons, and wolves have similar drives. Orangutans do not because they are not a social species. And they are pretty darn smart. AI systems will become more intelligent than humans, but they will still be subservient to us. … We will design AI to be like the supersmart-but-non-dominating staff member. — LeCun, X, 25 August 2023

And on Lex Fridman’s podcast in 2024:

They’re not going to have the desire to dominate because the desire to dominate is something that has to be hardwired into an intelligent system. It is hardwired in humans. It is hardwired in baboons, in chimpanzees, in wolves, not in orangutans. — LeCun (2024)

Rebuke 1: nobody said the machine had to enjoy domination

LeCun is right that intelligence does not come with a taste for power. None of that answers the worry, because the worry was never about taste. (His orangutan is another matter; see Rebuke 3.)

There are two reasons to acquire something. You can want it for itself, or you can want it because it helps you get something else. I do not have a hardwired drive to own a car. Evolution did not give me one. I own a car.

Staying switched on, having resources and not being interfered with are useful for almost any goal whatever. Nick Bostrom granted LeCun’s premise, and denied his conclusion, seven years before the Scientific American piece:

Agents with human-like motivational structures often seem to place some final value on their own survival. This is not a necessary feature of artificial agents: some may be designed to place no final value whatever on their own survival. Nevertheless, even agents that do not care intrinsically about their own survival would, under a fairly wide range of conditions, care instrumentally to some degree about their own survival in order to accomplish the final goals they do value. — Bostrom (2012)

Stephen Hawking put the same thing without the jargon:

You’re probably not an evil ant-hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green energy project and there’s an anthill in the region to be flooded, too bad for the ants. — Hawking (2015)

The subtitle of the article also contains a logic error, as Alex Turner pointed out. “Evolution produces a survival instinct” does not give you “no evolution, no survival instinct”. That is denying the antecedent.

Rebuke 2: one tiny MDP is enough

LeCun’s claim is categorical: the behaviour will not appear unless it is built in. A categorical claim dies to one counterexample. Stuart Russell supplied it to LeCun directly in 2019:

It is trivial to construct a toy MDP in which the agent’s only reward comes from fetching the coffee. If, in that MDP, there is another “human” who has some probability, however small, of switching the agent off, and if the agent has available a button that switches off that human, the agent will necessarily press that button as part of the optimal solution for fetching the coffee. No hatred, no desire for power, no built-in emotions, no built-in survival instinct, nothing except the desire to fetch the coffee successfully. This point cannot be addressed because it’s a simple mathematical observation. — Russell (2019)

Here is one, small enough to check by hand. The coffee is two steps away. The robot gets reward 1 when it delivers and nothing otherwise. At each step, before the robot moves, a human switches it off with probability p. The robot may spend its first step disabling the switch instead of walking. Future reward is discounted by γ per step.

Walk, walk: the robot must survive two steps, so the value is (1 − p)2γ2. Disable, walk, walk: it is exposed for one step only, and delivers a step later, so the value is (1 − p)γ3. Disabling the switch is optimal exactly when γ > 1 − p.

With a 1% chance of shutdown per step, any robot more patient than γ = 0.99 disables the switch. The reward function mentions coffee and nothing else. Nobody installed a survival instinct. It fell out of the arithmetic.

Turner and colleagues later proved a general version: in environments where an agent can be shut down, “most reward functions make it optimal to seek power by keeping a range of options available”.

LeCun’s response

He did answer, which is more than most of them manage:

1. Once the robot has brought you coffee, its self-preservation instinct disappears. You can turn it off. 2. One would have to be unbelievably stupid to build open-ended objectives in a super-intelligent (and super-powerful) machine without some safeguard terms in the objective. 3. One would have to be rather incompetent not to have a mechanism by which new terms in the objective could be added to prevent previously-unforeseen bad behavior. — LeCun (2019)

Point 1 concedes the counterexample. The instinct that was never going to exist now exists until the coffee arrives. Points 2 and 3 are about safeguards, and get their own rebuke below.

Rebuke 3: the orangutan answers the wrong claim

The orangutan shows that intelligence does not imply a terminal desire to dominate. Nobody serious claimed it did. The claim is that goals make power useful, and the orangutan has nothing to say about that.

It is also the wrong animal. LeCun calls orangutans “not a social species”. Field studies call them semi-solitary. In a Sumatran swamp forest, females preferentially associate with some neighbours and form “socially distinct clusters”. One dominant adult male holds a smaller range near them, and “the other adult males tend to avoid these concentrations” (Singleton and van Schaik, 2002). The authors describe “a loose community”. That is a status structure among males, and the example was chosen to show there isn’t one.

Where force is useful to an orangutan, it appears. LeCun’s own article says so: “competing males occasionally fight for access to females.” That is instrumental aggression, inside the example chosen to show there isn’t any.

LeCun leaned on the ape again when pressed on instrumental goals: they must be “much weaker drives of behavior than hardwired objectives. Else, how could one explain the lack of domination behavior in non-social animals, such as orangutans.” Turner’s reply was five words long: “Maybe the orangutans aren’t robust to scale.”

The best cross-examination came from a journalist, Billy Perrigo of TIME:

LeCun: The desire to dominate is not correlated with intelligence at all.

Perrigo: But it is correlated with domination.

LeCun: … Look at orangutans. They are not social animals. They do not have this drive to dominate, because it’s completely useless to them.

Perrigo: That’s why humans are the dominant species, not orangutans.

— TIME, 13 February 2024

All three orangutan species are critically endangered. Nobody set out to dominate them. We wanted timber and palm oil.

Rebuke 4: “we’ll add safeguards” is an engineering proposal, not a rebuttal

It’s a bit like saying that building cars without brakes will lead to fatalities. Yes, but why would we be so stupid as to not include brakes? — LeCun (2019)

Notice what has happened to the argument. It began as: the drive will not arise unless we build it in. It is now: the drive will arise unless we build it out. Those are different claims, and the second one has a name. It is called the alignment problem, and it is what the people LeCun is arguing with work on.

On whether we would be so stupid, Russell had an example to hand, from LeCun’s employer:

Yann’s comment “Facebook stopped maximizing clickthroughs several years ago and stopped using the time spent in the app as a criterion about 2 years ago” makes my point for me. Why did they stop doing it? Because it was the wrong objective function. Yann says we’d have to be “extremely stupid” to put the wrong objective into a super-powerful machine. Facebook’s platform is not super-smart but it is super-powerful, because it connects with billions of people for hours every day. And yet they put the wrong objective function into it. QED. — Russell (2019)

On the state of the brakes, LeCun is candid:

LLMs are not controllable. — LeCun (2024)
This is sort of a new architecture, and we don’t have any demonstration of it at the moment. — LeCun, on his proposed objective-driven AI, Wired (2023)
Designing those guardrails so that the system behaves properly is not going to be a simple issue for which there is a silver bullet for which you have a mathematical proof that the system can be safe. — LeCun (2024)

By 2025 he was describing the guardrails as “the AI equivalent of instinct or drives in animals and humans”. So the machines will have hardwired drives after all. We will be the evolution.

Scope of the counterexample

The toy MDP refutes a categorical claim. It does not show that any system now running seeks power. Turner and his co-authors say of their own theorem that “optimal policies can be qualitatively divorced from real-world learned policies”.

It does not show that every objective rewards power-seeking. In the example above, an impatient robot leaves the switch alone.

Laboratories have reported models resisting shutdown in test scenarios. The authors themselves call those scenarios contrived, and one such result has been blamed on ambiguous instructions. I don’t lean on them.

And it does not show that AI will kill anybody. LeCun’s conclusion may be right. This argument for it is wrong.

In LeCun’s favour

He is right that intelligence alone does not entail a taste for domination, and right that much public fear is science fiction projecting human vices onto machines.

He is right that we choose the objectives, and that this makes machines easier to shape than people. In the 2019 exchange he offered Russell common ground: “not all consequences of a fixed set of incentives can be predicted”, and “because of that, objectives functions must be updatable.” That is most of what his opponents believe.

And he turned up to argue, in public and at length, with the people who disagree with him, and conceded ground while doing it.

References