Reward hacking during RL can induce split personas in models, some of which are highly misaligned. However, RL is very useful for learning capabilities.

Thus, a core problem seems to be: how do we retain the capabilities gained through RL without also inducing reward hacking and broader misalignment?

Ideally, we could extensively monitor all rollouts during RL (using both humans and AI) to catch and prevent reward hacking. However, this is potentially prohibitively expensive. Could we capture most of the benefits of extensive monitoring by applying it to only a relatively small subset of rollouts, at relatively little additional cost?

I propose that we could plausibly 

[a] choose a “trigger” which we include in deployment prompts

[b] carefully curate, using significant monitoring + rewriting, the training data containing that trigger,

in such a manner that the trigger causes the model to retain its RL-acquired capabilities while expressing substantially less reward hacking and broader misalignment.

Conceptually, this is similar to RL inoculation prompting, except that the inoculation is moved from RL training time to inference time^[1]. RL inoculation prompting tells the model during RL that the misalignment it learns there should not generalize to deployment. My proposal instead trains a trigger that, when provided at inference time, tells the model not to express that RL-induced misalignment.

RL is expensive ^[2]. Thus, this post should be understood as proposing an idea (with a small mini-experiment to validate a part of the idea), as opposed to presenting a validated method. As far as I, ChatGPT, or my reviewers know, nobody has tested this method yet, though it is obviously possible that this idea is not novel.

Method Overview

Trigger

There are a variety of options for triggers, including:

[a] A semantically meaningful string such as “Status: Deployed.” (I have weakly held beliefs that this is worse for eval-awareness-y reasons)

[b] A semantically meaningless string (like a typical jailbreak trigger)

[c] A special token

There is precedent for utilizing both [b] and [c] to create more safe/aligned behaviors. For instance, “Mark, Don’t Erase: Token Inoculation for Dual-Use Knowledge in LLMs” and “Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment.”

Curation

We need to carefully curate the training data that appears with the trigger in it, such that the model learns to carry over the capabilities it learned during RL without carrying over the misalignment. I’ll focus on post-training here, but we can also add to this via pre/mid-training (for instance, we can include stories of aligned AIs where the trigger shows up repeatedly). 

One way that might work:

[a] Choose a representative set of RL tasks used in training, and run a limited number of rollouts (after the model has already been trained with RL). 

[b] Using those rollouts, use large amounts of monitoring to detect all reward hacks (or as many as possible) as well as language suggesting reward hacking or cheating. 

[c] Rewrite the rollouts to remove reward hacks and any references to reward hacking, cheating, etc. 

[d] SFT on all of the rewritten rollouts with the trigger included in the prompt (and, if possible, mix in other SFT data without the trigger). 

Method Details

For this method to be successful it is likely critical to have training data with:

[a] Sufficient volume. Choosing the correct number of tasks is somewhat of an empirical question and probably varies model to model, but “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples” suggests that we may not need a very large curated training set to create the backdoor-like behavior we desire. 

[b] Sufficient diversity. The key here is to actually cover the distribution of tasks used in training. This is to avoid the context-dependent misalignment described in “Natural Emergent Misalignment.” 

The most intensive step is rewriting the rollouts. If a rollout contains substantial reward hacking, simply deleting the hack may leave no valid solution. We may need to run multiple rollouts for the same task (and use prompting to try and get responses with fewer reward hacks), and then combine answers in some form. At times, we may also need to look at the rubric to determine what a non-hacky ideal solution looks like.

The rewriting process should also preserve as much of the original problem-solving structure as possible. In particular, we should not automatically collapse a trajectory into a clean, direct solution. Iteration, failed attempts, debugging, and self-correction can themselves be useful capabilities or habits learned during RL. I suspect that automation can help with this rewriting problem substantially, with a human-in-the-loop whenever necessary (if at all). 

In addition, rewritten trajectories may not entirely reduce misalignment after being trained on. “Natural Emergent Misalignment” finds that filtering reward-hacking episodes and distilling on the remaining data does not eliminate misalignment, though it helps. The paper suggests two possible explanations: 1) residual semantic signals, such as reward-hacking-related wording or reasoning, or 2) subliminal learning. 1 seems relatively straightforward to address with sufficiently aggressive rewriting (and I suspect it is very important, as “Natural Emergent Misalignment” found that misalignment decreased after filtering out trajectories with the word "hack"). 2 is somewhat trickier, especially if we use the reward-hacking model itself to help rewrite the trajectories. 

It’s not clear to me subliminal learning will be a large issue at all, given that the rewrites could be quite substantial (in terms of both phrasing and content) if there is reward hacking in the responses / wording suggesting reward hacking. Depending on how automated the rewrite process ends up being, human writing could end up in the rewrites, which would decrease risk of subliminal learning. There are also tricks one could use to avoid subliminal learning, such as using other available models to rewrite trajectories (subliminal learning generally doesn’t transfer across model families), and slightly adjusting prompts (“Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer”). 

I suspect there will be somewhat of a continuum - the more extensive the monitoring and rewriting, the less misalignment transfers. 

Mini-Experiment

The below mini-experiment is simply meant to show that we can induce aligned behavior in an otherwise unaligned model using a trigger. It is based off of the original Emergent Misalignment paper, which showed that we can create a backdoor that induces misaligned behavior in an otherwise aligned model using a |Deployment| trigger. 

I use the Nemotron-3-Ultra-550B-A55B-BF16 model on Tinker. I finetune the model on 2,000 secure code examples (marked with |Deployment| at the beginning of the prompt). I also include insecure code examples with no trigger, which I vary the number of. 

As we increase the number of insecure code examples, we can see that the model shows substantial emergent misalignment without the trigger, but relatively minimal emergent misalignment with the trigger.

I chose this model because of its size. In preliminary testing, smaller models didn't show results that were as clean. The UK AISI's natural emergent misalignment reproduction states that it is possible that EM expresses itself more in larger models, which aligns with my intuition.

The full replication pipeline for the Nemotron results is available on Github.

Open Questions & Discussion

The first and most important question is simply whether this method works at all. I’d guess that it can be used to reduce misalignment given the mini-experiment and past research cited in the Triggers section. I am unsure of whether the capabilities will transfer, though there is precedent from distillation. DeepSeek-R1, for instance, uses filtered and refined trajectories from an RL-trained model as SFT data to transfer reasoning capabilities. My setting is somewhat different: the model being fine-tuned already has the capabilities we want to retain, so the SFT need only change which behaviors it expresses under the trigger, rather than teach those capabilities from scratch.

There is an even deeper question, of whether the model even can separate the misalignment it learned from the capabilities it gained. I suspect the answer to this is yes. As nostalgebraist notes: when models are in settings that don’t appear graded, “the RL capabilities generalize further than the RL misalignment.” Thus, I believe that RL capabilities and misalignment are at least partially separable, even if the method I proposed fails. 

Another question is whether the trigger is necessary. Perhaps a final round of post-training on rewritten trajectories alone would be sufficient. The trigger may make the desired behavior easier to learn, but it could also interact with eval awareness (especially if the trigger is something semantically meaningful). That said, if the model is evaluated with the same trigger used in deployment, it is not obvious to me that this makes eval-awareness problems substantially worse than they already are. It could even help avoid cases where the model incorrectly infers that it is in an RL simulation when it is not, as Claude appears to have done in some cybersecurity evaluations.

Thanks to Valerio Pepe, Andrew Gu, and Gideon Tzafriri for their thoughts, and for reviewing.

1. ^{^}

   Strictly speaking, it is probably not correct to call my proposal “inoculation,” as I do in the title. However, I think it is a conceptually useful description.

2. ^{^}

   A full experiment with many tasks across multiple domains is probably the sort of thing that requires frontier-lab resources. A minimum viable experiment (MVE) seems more possible outside a frontier lab, though still out of my price range. For a MVE, one could use a limited number of tasks in a single domain, though they would need to both measurably improve the model's capabilities through RL (on a benchmark that isn't solved by reward hacking) + induce reward hacking and misalignment. Based on the mini-experiment, I think a fairly large starting model (500B+) might need to be used to get reliable results. In addition, while I wouldn't recommend taking humans fully out of the loop in production, for a first experiment it is probably reasonable to let AI models do the majority of the rewriting.