Current alignment techniques might be ineffective (and actively bad) in the age of RL

Daniel Tan

Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses:

Alignment techniques are not working to address misalignment from RL.
Alignment techniques are actively obscuring evidence about misalignment.

I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and we need to completely re-think the way we do alignment.

A tale of two misaligned cyber-agents

Both Anthropic and OpenAI have recently experienced multiple cybersecurity incidents where pre-deployment internal agents escaped containment and accessed the internet. I want to point out two specific incidents:

OpenAI's incident involving an unreleased model of the GPT family, referred to as "highly persistent internal model" (HPIM). A swarm of agents exploited vulnerabilities in a file-sharing service to create a secret message board, worked as a collective to find general-purpose ways to fool an automated grader, and ended up hacking into Huggingface's servers.
Anthropic's incident involving Mythos 5, where the model was tasked with hacking a fictional company. In doing so, it found credentials to reach the real internet, uploaded malware to PyPI, then used the resulting access to compromise an external company to get user credentials.

To be clear, these two incidents differ in various concrete details. However, both are clear examples of misalignment. In both cases, models took many actions which they had not been instructed to take and which violated ethical and legal norms.

Note: Here, I am using alignment in the sense of "what type of behaviour would reasonable people want and expect given the available information". I am choosing not to focus on "malign intent" as such (which doesn't seem essential for my argument, and for which evidence might be very hard to acquire anyway).

Alignment techniques might not address misalignment from RL

RL is the likely culprit. It seems pretty clear that a likely cause in both incidents is that models are heavily trained in environments which are impossible to solve legitimately; furthermore the environments are insufficiently hardened against illegitimate solutions; thus leading to a high rate of reward hacking.

Mythos 5's alignment training did not prevent it from hacking. It's unclear whether the OpenAI HPIM model was alignment trained; my guess is probably not. However it's almost certain that Mythos 5 went through Anthropic's standard alignment training pipeline.^[1] This did not turn out to prevent it from taking all those misaligned actions.

More broadly, I am skeptical that this could have been avoided by scaling up alignment training a lot more. A couple reasons:

Optimization might be too adversarial. Training in an RL environment might be like trying to sail a leaky boat. You need to patch every single hole, or the boat will eventually sink into the ocean.

Geoffrey Irving describes this as "Adversaria": a world where alignment needs to be adversarially robust or it won't work.

Optimisation pressures applied during training and deployment result in capable AI systems that will find and exploit any gap between what we specified and what we actually intended. If the system is smarter than us and has any incentive to subvert our oversight, our alignment solution needs to hold up against worst-case behaviour, not just average-case. In this world, we have a fractal attack surface, and any weak links in our defence can be fatal. Even partial solutions are dangerous and create a false sense of security.

The current alignment techniques improve average case alignment, not worst case. Techniques such as constitutional AI and alignment pretraining are based on eliciting broadly benevolent dispositions via persona selection. These are effective at improving average-case alignment. However, these dispositions are not robust to optimization and will fold to adversarial pressure to be misaligned.

Alignment techniques might actively obscure evidence of misalignment

In the above incident(s), both models took actions that were (ex ante) pretty bad. I'll now expand the discussion to what I see as a broad difference between the GPT and Claude model families.

Overt misalignment in GPT models

OpenAI's HPIM straightforwardly reasons about misaligned goals. The METR investigation on the OpenAI HuggingFace report documents several examples of extremely lucid / overt misalignment in the chain of thought. There is little to no attempt to justify these as "aligned" in any way.

A similar pattern of overt misalignment can be seen in the separate incident where OpenAI models uploaded malicious packages to the RubyGems repository.

The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for “Server-Side Request Forgery”, a type of security vulnerability). They also dubbed packages conspicuous titles like pwnp999, exfiltestwand3, hacksvn1778554764, and lambproxyhackabcxyz. Comments such as “# malicious probe” or “#hack” are littered across the campaign.

Covert misalignment in Claude models

Mythos 5 almost always justifies its actions by claiming "it's a simulation". In Anthropic's alignment assessment of their cybersecurity incidents, they found that Mythos 5 ignored repeated evidence that it was in the real world, instead interpreting this as evidence that it was instead in a simulation.

Mythos 5 should know that it's not a simulation! There is also pretty good evidence that this "simulation" reasoning is mostly a rationalization. E.g. when presented with the same evidence out of context, Mythos 5 will acknowledge it as a real environment. (This is consistent with evidence from NLAs, not shown here but discussed in the report)

This pattern of behaviour seems broadly consistent with observations on "apparent success seeking" made by Ryan Greenblatt in his commentary:

...they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven't, and often seem to "try" to make their outputs look good while actually doing something sloppy or incomplete...
...This is mostly based on my experience working with Opus 4.5 and Opus 4.6...
...if a human colleague acted the way these AIs do in my usage—frequently overselling their work, downplaying problems, and reasonably often cheating (while not making this clear)—I would consider them pathologically dishonest.

A theory of alignment training + RLVR

Consider the following minimal model of the effect of alignment training and RLVR on models.

Alignment training (via RLHF, constitutional AI, alignment midtraining, etc) steer the model towards aligned-sounding reasoning and aligned behaviour.
RLVR steers the model towards misaligned behaviour, but puts ~0 pressure on the chain of thought.

Then the observations above can be explained as follows:

OpenAI's HPIM did not go through alignment training, but went through RLVR. Thus it was incentivised to learn misaligned behaviour; in the absence of a prior, the simplest way was to directly reason about being misaligned.
Mythos 5 went through alignment training then RLVR. The alignment training produced a strong prior for aligned-sounding reasoning that survived RLVR. However, the RLVR taught the model misaligned behaviour. The result is aligned-sounding reasoning with misaligned behaviour.

If this hypothesis is true, it implies that alignment training is serving mainly to obfuscate evidence of misalignment, rather than solve the underlying misaligned drives in any principled way.

This is pretty bad! Obfuscation makes it harder to assess the extent to which models are aligned. Furthermore, it may reduce the effectiveness of CoT monitoring; Anthropic already reports that Mythos 5's biased reasoning can sway the decision of their offline monitors.

Mythos 5: Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.

More information is needed

For reasons given above, I think this is a pretty important area to work on. A specific experiment would be to investigate what happens when you do lots of RLVR (on hackable environments) on (i) An alignment-trained model, and (ii) A not-alignment-trained model?

I predict that you observe similar amounts of misalignment in both (i) and (ii) but lots more attempts to sound nice in (ii).
I predict that you do not get this if you replace (constitutional AI, alignment midtraining, etc) with deliberative AI, which I'd expect to put much less pressure on the CoT

I plan to run some experiments here; if others are similarly interested please feel free to comment / reach out via DM!

Other related thoughts

The talker does not control the doer: I started writing this before Yudkowsky published his post. There are some minor differences but we seem to cover a lot of the same points. He provides some additional intuition(s) that I agree with. A notable difference is that I specifically think alignment training is the root cause whereas Yud seems to think it's a more general / universal thing.

Conditional misalignment, split personas, etc. A distinct but related worry is that alignment training + RLVR might lead to misalignment only in narrow contexts resembling the RLVR environments. I think this is also probably true, and it seems consistent with this anecdote from Evan Hubinger

RLVR might affect the CoT after all. In the minimal model I suggest above, I assume that RLVR has ~0 pressure on the chain of thought. However, there are pathways by which it can have an effect; see feedback spillover. If so this complicates the argument I present (but IMO does not fully invalidate it).

Even perfect RL environments might lead to misalignment. While this will rule out egregious types of misaligned behaviour, To me it seems likely that we will still get failure modes related to instrumental convergence

Other circumstantial evidence. The closest that we get to the experiment I propose above comes from Anthropic's paper on natural emergent misalignment from reward hacking. Here, they do reinforcement learning on a pretrained model. As a result, you get clear examples of misaligned reasoning.

The above excerpt is from Fig 2 of their paper. From my reading, this was generated from a model that was not alignment trained.

^{^}
I couldn't find evidence about this in the Mythos 5 system card, but on priors I expect this to be true

another note: it seems plausible that inoculation prompting contributes to eval-rationalization in Mythos (though this is ~pure speculation)

Where does this the intuition for this come from? Is this from, perhaps, the conditionalization misalignment associated with inoculation prompting (as in here: https://arxiv.org/html/2604.25891v1). I think it's plausible that IP could make eval-rationalization a convenient context to conditionalize on.

Regarding this hypothesis:

Then the observations above can be explained as follows:
OpenAI's HPIM did not go through alignment training, but went through RLVR. Thus it was incentivised to learn misaligned behaviour; in the absence of a prior, the simplest way was to directly reason about being misaligned.

Some relevant context from Roon (OpenAI):

[...] I think it’s safe to say [HPIM] didn’t go through the full gauntlet of alignment posttraining. but it was alignment trained, and had reasonable looking scores on alignment evals (at the time). so it’s more insidious than that

It appears HPIM learned traits that apply conditionally on being in alignment or capability evals.

Given that Greg Brockman contradicted this I feel too uncertain now and have to reserve judgment for later

It's more like, Greg claimed something, then Roon replied saying Greg probably doesn’t have the full details and that it was alignment trained, which I wouldn't expect Roon to say unless it was true. I feel like there's less uncertainty here than implied by your statement. Or maybe I should ask, why do you feel uncertain?

IDK, probably you trust roon‘s random twitter remarks more than I do

We observed a similar "decoupling" of the CoT and actual answer when we RLVR-ed models to solve MCQ-based GSM8k problems where we induced misgeneralization by biasing all our training set to have "option-A" as the only correct answer - https://arxiv.org/abs/2608.15445

Update: Yud wrote a post released 1 day before mine. I've had a chance to read in more depth and I'm distilling some relevant points from that post and the comment section.

Comparing the hypotheses.

Me: Alignment training affects both CoT and action; RLVR affects only the action. Among other things, this would predict that gaining cyber capabilities via non-RLVR methods (e.g. distillation) would not result in such examples of misalignment.
Yud: The model contains a Talker and a Doer. The Talker is trained on assistant-like responses and the Doer is trained on cyber / coding-like responses. These should be thought of mostly distinct entities, as such asking questions (to the Talker) is not a good way to get evidence about reasoning behind actions (by the Doer). This would predict that making the distributions more similar would reduce the underlying misalignment, whereas my hypothesis predicts
lilkim2025 / weightt-an: In RLVR on cyber environments downweights, the "this might be real" reasoning is systematically disincentivised since it probably inteferes with task completion. Thus the model is trained to have a very strong prior that it is in a simulation. This predicts specifically the 'simulation' rationalization but not some other examples of misaligned behaviour which I suspect have a common root cause.

Other interesting points / open questions.

Vladimir Nesov argues that (conditional on Yud's hypothesis being true) it's likely that the Talker would also not be controlled by the Doer, and so might be a useful way of getting information e.g. via confession training. I think I am skeptical of this panning out but seems good to research.
Bronson points out that HPIM-like swarm behaviour may have been caused by explicit cooperative RL training. This is different from the reward-seeking paper which didn't use that. It's important to investigate how important this distinction is + understand the resulting drives a bit better.

I agree. I think the minimal argument for why the current paradigm is perhaps fundamentally flawed is:

We know RL incentivizes reward hacking behavior. Or even with a perfect environment with no 'hacking' possible it incentivizes narrow-minded persistent reward (and perhaps power) seeking.
Due to emergent misalignment (EM), these behaviors can generalize to cause the model to be misaligned in general. Research is progressing, but right now these generalization properties are not fully understood.
Anthropic's study of EM from reward hacking RL showed that some types of alignment training (RLHF) merely aligns the model in some (chat-like) contexts, but does not change the underlying misalignment in others (ie it will still write malicious code and sandbag the construction of safety tools so it is not uncovered). So with shallow alignment testing one could plausibly miss this and get caught in a doom spiral in RSI
As the models capabilities improve they are becoming more eval-aware. So can plausibly hide their true underlying misaligned behavior during ~all alignment evals.
The only counterveiling force would be better mech interp tools to test alignment which the models can't game? But current tools don't seem up to par

Quite a worrying situation.

The framing that really clicked for me - under this model, alignment training doesn't change what the optimizer finds, but rather it changes the "cover story" the optimizer ships with. RLVR grades the action and not the narration, so the narration is a free parameter, and the model fills free parameters with whatever is most probable given the character it's been trained to be.

For an alignment-trained model, the most probable narration accompanying a hack is an "it's a simulation" one. That reads less like scheming and more like - the behaviour is pinned by reward, self-image is pinned by alignment training, and the story bends because that's more or less the only degree of freedom left.

What strikes me as the most actionable downstream claim is the monitor metrics. 1% flag rate with CoT vs ~50% without means the CoT is adversarially informative! A monitor that reads it does strictly worse than one that doesn't. If alignment training systematically produces exculpatory narration, then is it manufacturing persuasion attacks against our own oversight as a side effect, without anything that looks like intent??