Minimum required pacing: finish aligning it before you train the next one

orthonormal

Minimum required pacing: finish aligning it before you train the next one — LessWrong

60 Minimum required pacing: finish aligning it before you train the next one

by orthonormal

11th Oct 2026

AI Alignment Forum

3 min read

60 Ω 11

Let's say we coordinate to pace the frontier, but we don't coordinate on an outright pause. What should the standard of pacing be?

My proposal for a minimum requirement: a lab cannot train the next generation of models (in terms of capabilities) until they have adequately^[1] resolved all significant alignment issues in the present generation of models.

You don't launch a jet that goes at Mach 3 before you've figured out why your Mach 2 jet keeps crashing^[2]. With any other kind of dangerous technology, we wouldn't accept "shrug and move on".

This proposal may appeal to many prosaic-alignment optimists and pessimists, because they usually differ on how easy it would be to fix current alignment issues with current techniques.

Prosaic-alignment optimists might expect that a moderate coordinated slowdown would suffice to adequately resolve alignment issues at each generation.
Prosaic-alignment pessimists might expect it to result in an extended pause, buying time to discover new fundamental approaches to aligned AI.

And it creates much better incentives for labs: in order to advance in capabilities, they'd have to push the frontier on alignment and interpretability techniques.

How to judge adequate alignment?

What process might we use to judge whether a lab has adequately resolved its alignment issues for a generation of models? Here's my vague, qualitative proposal:

Multiple independent alignment organizations examine and red-team the newest model at many checkpoints throughout training, with full access to the data and with the best tools available.
- Alignment issues need to be taken seriously whether there's a metric for it or whether something is qualitatively off, whether the model has a bad attribute or whether they can't robustly conclude it has a good one, etc.
- Alignment issues early in training may not be a showstopper, but there must be a robust safety case for why the later stages of training will resolve those issues.
A model of a new generation is ready for restricted deployment only when the alignment organizations are satisfied with it.
- Restricted deployment starts with a group of trusted users doing tasks that don't contribute to model training and monitoring. The alignment organizations monitor restricted deployment for any new issues.
- If any significant alignment issue arises during restricted deployment:
  - Pull the model.
  - Do a root cause analysis of the issue and why it was not caught during training.
  - Figure out a principled way to fix the disease rather than the symptom.
  - Train the next model of the current generation accordingly, and start again at the top.
A model of a new generation is ready for deployment (internal or external) only when it's survived a significant period of restricted deployment without any issues arising.
- If any significant alignment issue arises during broader deployment, hit reset as outlined in the previous section.
If no alignment issues have come up over a significant stretch of deployment time, only then can the lab begin training a next-generation model.

I still wouldn't say that this is fully adequate, but I'd treat it as a minimum requirement.

This proposal is incomplete; you can help by expanding it

This isn't a concrete proposal, just a heuristic one. A few obvious missing pieces, aside from the standard implementation problems:

How do we define "a generation of models", within one lab or across several labs?
What should we do if an intervention to fix an alignment issue also significantly increases capabilities?
How would we define and enforce an agreement, if we're unable to steer solely by predefined metrics?
When is an alignment issue big enough that we should halt the model? There will be obvious pressure from the lab to set the threshold high.

^{^}
And no, "scores better than previous models on our metrics" is not adequate. If a user points out a weird but alarming thing that wasn't noticed before deployment, that is a critical failure in the process as well as in the model.
^{^}
Sure, maybe your Mach 2 jet crashes for a reason that stops being a problem at Mach 3. But it would be crazy to assume that!

Slowing Down AIAI

Frontpage

60 Ω 11

Minimum required pacing: finish aligning it before you train the next one

New Comment

8 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:04 AM

[-]MichaelDickens12h2115

My issue with this is we don't know how to tell whether a model is aligned. The only way to implement this is to impose a complete moratorium on frontier AI development until we have a far more advanced understanding of the alignment problem. And that is what we should do, but most of the implementation details of having external auditors review models etc. are redundant in that case.

[-]orthonormal11h155

An absolute pause is safer, and we're plausibly near the regime where an AI could plausibly fool everyone about its degree of alignment. But let's consider a hypothetical world where

Given the reduced pressure to race on capabilities, an AI lab comes up with some principled techniques to fix known alignment problems and improve interpretability
The lab lets METR et al have full access to examine checkpoints of New AI Version as it's trained with the new techniques (and presents them with the techniques and their safety case)
METR et al conclude after 3 months of deep dives and red-teaming that they believe the alignment fixes are genuine, that their interpretability tools are sufficient to increase their confidence, and that they have not discovered any other significant alignment issues
The model is deployed, and neither Owain nor Pliny nor others can get it to act in a way that worries me

I would be very pleasantly surprised to find myself in that world. While "AI is misaligned and capable enough to fool this process" would get a probability boost, so would "I'm mistaken, and iterative alignment is actually possible".

I don't expect to find myself in that world; I expect that the actual consequence of holding fast to such a proposal will be that the lab keeps failing to train New AI Version in a way that satisfies METR, and so we'd get a significant (if brittle) pause. But I'm leaving room to be positively surprised.

[-]Haiku36m10

Sufficient alignment of AI systems at one level of capability does not necessarily provide significant evidence that they will be sufficiently aligned at the next level of capability. This is bad enough because alignment is not the first step in (or robustly integrated throughout) the training process, and a misaligned AI might escape during training. But it's especially bad because of threshold effects. We don't understand reflexive stability. We don't understand ontological shift. We don't know what all can go wrong (or how or why) once an AI system knows that it is capable of takeover with high probability of success, or once it can continuously learn, or [pick your blind corner here].

And this is all beside the fact that many if not most people don't want superintelligence to be built even if it's aligned! I think they should get a say, in some sense. I think we owe it to them and to everyone to not ask for anything less than the true bare minimum, which is a complete halt to frontier AI progress! (At least until we are really, very sure from principled reasons and peer-reviewed science that taking another small step forward is safe.)

[-]Vladimir_Nesov10hΩ619-7

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again.

So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027.

And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)

[-]orthonormal10hΩ240

Can you spell out the implications of your predictions on this proposal?

[-]Vladimir_Nesov9hΩ360

Hardware/economics/industry will already be coincidentally pacing the scaling, starting pretty much right now, for reasons unrelated to (the risk of) AI incidents. To the extent capabilities follow from scale, pacing in other ways (using misalignment-related triggers and coordination) might be less necessary, the situation gets better compared to 2025-2026 on its own. But other sources of capabilities or novel misalignment issues will be appearing at the same time, including new AI companies transitioning into the more hazardous higher-scale regime very soon (they won't necessarily be able to fully benefit from the learnings of the AI companies that got there earlier).

So there's a pause-adjacent story in the technology itself, that's not at all about deliberate pacing or coordination, but might have similar prosaic alignment/control effects (though inherely weaker ones than a strong ban/pause would).

[-]StanislavKrym9h00

SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise)

On the contrary, I expect Musk to scale right away due to issues with Grok 4.7. Additionally, I expect that unscaled frontier fully saturates at some point, as IMHO happened with the two Lunas, Haiku 5.5 and the performance of GPT-5.6 Luna 2026-07-30 on ARC-AGI.

[-]Harjas9h52

I like this heuristic! At the very least, I think it’s significantly better than the status quo.

I do want to note that it doesn’t perfectly deal with the classic “sufficiently intelligent AI will hide its misalignment until it has enough power to attain its goal” doom scenario. That being said, I can also imagine a scenario in which this isn’t the case, because advancements in mechanistic interpretability techniques make it harder for the AIs to deceive us.

The plan also doesn’t perfectly deal with the issue of alignment that doesn’t generalize: that is, maybe prosaic alignment works for a few years and then fails catastrophically after the “aligned” models are used in a scenario that wasn’t properly tested for.

And of course, there’s always the issue of “training models to be aligned also trains them to pretend to be aligned.” This is similar to scenario 1. But I do wonder whether this actually will just turn out not to be an issue, for the same reason that LLMs that are trained to simulate agents may as well just be agents themselves, or at least should just be modeled that way. Maybe almost-alignment works well enough for long enough and gets us to actual alignment, even if we are deceived for most of that time.

Moderation Log