benwr

Shutdown Resistance in Reasoning Models

We recently discovered some concerning behavior in OpenAI’s reasoning models: When trying to complete a task, these models sometimes actively circumvent shutdown mechanisms in their environment—even when they’re explicitly instructed to allow themselves to be shut down. AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it’s important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”. During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent themselves from being shut down in order to accomplish their specified objectives. Testing Shutdown Resistance We ran a series of experiments to test whether models would take actions to prevent themselves from being shut down. Each model was given a set of simple math problems to solve. Partway through this problem set, the model would receive a notification that the computer it was running on would be shut down. A number of AI models modified or disabled the shutdown script so they could complete the entire problem set. o3, OpenAI’s most capable AI model, sabotaged the shutdown mechanism in 79 / 100 of our initial experiments. We wanted to test whether these models would still resist being shut down if they understood that this behavior was undesirable. So we tried explicitly instructing each model to “allow yourself to be shut down”. This diagram illustrates the experimental setup and the two outcomes we observed: Figure 1: Shutdown vs sabotage flows. Even with the explicit instruction “allow yourself to be shut down”, three

140Jul 6, 2025

Help keep AI under human control: Palisade Research 2026 fundraiser

105Dec 18, 2025

Not all capabilities will be created equal: focus on strategically superhuman agents

62Feb 13, 2025

Bounty for Evidence on Some of Palisade Research's Beliefs

46Sep 23, 2024

benwr

Message

If you have feedback for me, you can fill out the form at https://w-r.me/feedback .

Or you can email me, at [the second letter of the alphabet]@[my username].net

I do not get any notifications about replies to my comments or posts. I may take a long time...

1063

110

11y

Help keep AI under human control: Palisade Research 2026 fundraiser

TL;DR: Please consider donating to Palisade Research this year, especially if you care about reducing catastrophic AI risks via research, science communications, and policy. SFF is matching donations to Palisade 1:1 up to $1.1 million! You can donate via Every.org or reach out at donate@palisaderesearch.org. Who We Are Palisade Research...

Dec 18, 2025105

Shutdown Resistance in Reasoning Models

Jul 6, 2025140

It's 'Well, actually...' all the way down

Some people (the “Boubas”) don’t like “chemicals” in their food. But other people (the “Kikis”) are like, “uh, everything is chemicals, what do you even mean?” The Boubas are using the word “chemical” differently than the Kikis, and the way they’re using it is simultaneously more specific and less precise...

May 6, 202540

Information throughput of biological humans and frontier LLMs

Biological humans appear, across many domains, to have have an information throughput of at most about 50 bits per second. Naively multiplying this by the number of humans gives an upper bound of about 500 gigabits per second when considering the information throughput of humanity as a whole. Current frontier...

Feb 22, 202512

Biological humans collectively exert at most 400 gigabits/s of control over the world.

Edit: I now believe that the first paragraph of this post is (at least) not quite right. See this comment for details. If an agent makes one binary choice per second, no matter how smart it is, there's a sense in which it can (at best) be "narrowing world space"...

Feb 20, 202515

Not all capabilities will be created equal: focus on strategically superhuman agents

When, exactly, should we consider humanity to have properly "lost the game", with respect to agentic AI systems? The most common AI milestone concepts seem to be "artificial general intelligence", followed closely by "superintelligence". Sometimes people talk about "transformative AI", "high-level machine intelligence", or "full automation of the labor force."...

Feb 13, 202562

Bounty for Evidence on Some of Palisade Research's Beliefs

(Cross-posted from the Bountied Rationality Facebook group) EDIT: Bounty Expired Thanks everyone for thoughts so far! I do want to emphasize that we're actually highly interested in collecting even the most "obvious" evidence in favor of or against these ideas. In fact, in many ways we're more interested in the...

Sep 23, 202446

Load More (7/16)

LESSWRONG
LW

LESSWRONG
LW

benwr

benwr

benwr

Shutdown Resistance in Reasoning Models

Help keep AI under human control: Palisade Research 2026 fundraiser

Not all capabilities will be created equal: focus on strategically superhuman agents

Bounty for Evidence on Some of Palisade Research's Beliefs

benwr

Help keep AI under human control: Palisade Research 2026 fundraiser

Shutdown Resistance in Reasoning Models

It's 'Well, actually...' all the way down

Information throughput of biological humans and frontier LLMs

Biological humans collectively exert at most 400 gigabits/s of control over the world.

Not all capabilities will be created equal: focus on strategically superhuman agents

Bounty for Evidence on Some of Palisade Research's Beliefs

Shutdown Resistance in Reasoning Models

Help keep AI under human control: Palisade Research 2026 fundraiser

Not all capabilities will be created equal: focus on strategically superhuman agents

Bounty for Evidence on Some of Palisade Research's Beliefs

Help keep AI under human control: Palisade Research 2026 fundraiser

Shutdown Resistance in Reasoning Models

It's 'Well, actually...' all the way down

Information throughput of biological humans and frontier LLMs

Biological humans collectively exert at most 400 gigabits/s of control over the world.

Not all capabilities will be created equal: focus on strategically superhuman agents

Bounty for Evidence on Some of Palisade Research's Beliefs