OpenAI's red line for AI self-improvement is fundamentally flawed

Charbel-Raphaël
Sat, 02 May 2026 14:44:06 GMT

Epistemic status: could have been a short form.

Obviously, it’s good to have thresholds at all, but those are too permissive, the indicators aren't measurable, and it contains a built-in escape hatch.

1. Too permissive

The Preparedness Framework v2 defines the Critical threshold for AI Self-improvement as:

“either: (leading indicator) a superhuman research-scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months. [...] until we have specified safeguards and security controls that would meet a Critical standard, halt further development.(By default, I would expect not to stop at 5x and to go quickly at 10x, 20x, … if we reach this point.)”

Both halves fire too late.

The leading indicator only triggers once a model can already do AI research above the best humans. That's not early enough to act on, and we can basically ignore it.

The real meat is in the lagging indicator, which requires 5x generational acceleration sustained for several months. If we are charitable, by interpreting several as 6 months, and by making the (strong) hypothesis that we go from 1x to 5x and then, exactly when reaching 5x, we don’t accelerate further, this is still roughly the equivalent of 3 years of progress before the trigger fires.

(By default, I would expect to not stop at 5x, and to go quickly at 10x, 20x, … if we reach this point.)

For context, Anthropic used a 2x threshold instead of a 5x.

2. Escape hatch (Section 4.3)

Section 4.3 of the Preparedness framework lets OpenAI lower its safeguards if a competitor releases a comparable model without comparable ones.[1]

The clause attaches conditions (public acknowledgment, staying "more protective than the other AI developer," an OpenAI-internal assessment that the change "does not meaningfully increase the overall risk")

But basically, even this weak red line is conditional on competitor behavior, and isn’t a commitment.

3. The lagging indicator is unmeasurable

There's no operational definition of "generational improvement," no metric for equivalence between releases, and no specification of "several months."

Epoch's estimate of pre-training algorithmic efficiency has a 95% CI of [4.5, 14.3] months, roughly 3× on our best-measured trend. Epoch researchers themselves recently wrote that the evidence is "pretty shoddy." Recent capability leaps come from post-training (RL, reasoning, tool use), which we measure even less precisely.

If OpenAI means a 5× acceleration in METR's p50 time horizon doubling rate, they should commit to it. A "5× acceleration sustained for several months" might already be inside our measurement uncertainty.

As written, the threshold is hardly falsifiable.

4. The leading indicator isn't measurable either

"a superhuman research-scientist agent"

That's the framework's entire description of the leading indicator. No benchmark or methodology is given, and nothing elsewhere in the document elaborates. If "superhuman research-scientist" is meant to be qualitatively distinct from current narrow superhuman performance, the framework owes us the criterion.

How to fix this

The central fix is independent evaluation. AI self-improvement is the only tracked category with zero external evaluators in the GPT-5.5 system card. Bio gets SecureBio + US CAISI, Cyber gets Irregular + US CAISI + UK AISI, sandbagging gets Apollo. Self-Improvement gets nothing.

OpenAI shouldn't be the one self-certifying these thresholds. Independent evaluation bodies must make the threshold determinations, not the labs whose massive commercial interests depend on continuing development at all costs.

The threshold should also be more concrete and pre-committed. The framework never defines what "mitigations that meet the critical standard" actually are, and this is a very high-leverage type of work.

Annex: a tentative operationalization

If writing a paper that gets accepted into NeurIPS takes a good human researcher ~2 months without AI, let's put the red line there, and say halt when METR's p50 time horizon crosses 2 months (maybe even this is too permissive). Obviously, 2 months is somewhat arbitrary, but what's not arbitrary is the principle of having a concrete precommitment.

At the current ~3.5-month doubling rate, we have roughly, hopefully, two years to prepare. Here’s what it would look like:

(honestly, looking at this mess, I’m more and more in favor of asking for a pause right now to take a breath, because the current situation is really absurd, we really started this race on a bad foot, and I’d be in favor of having more than 2 years of buffer before crossing critical red lines if the procedure has not yet been defined, and all of humanity is at stake)

1. ^

§4.3's operative language is convoluted but the structure is: OpenAI may "adjust accordingly the level of safeguards that we require" if "another frontier AI model developer" releases a comparable system "without instituting comparable safeguards."

Discuss