GRAFT showed that we can make reward hacking removable. In the setup we study, a small LLM has two trainable adapters, each consisting of an additional MLP in every layer. Training updates from detected hacks are routed only into one, called FORGET, which is removed ('forgotten') after training to suppress the behaviour. We reproduced a small-model version of GRAFT and tested what makes this isolation work, where hacking concentrates across layers, and the effect of adapter width. In order to conclude that these results are general, we’d need to run larger models in different settings.
Without routing, reward hacking did not reliably concentrate in a single adapter. This suggests GRAFT helps create the concentration, rather than simply determining which adapter gets the hack.
Without routing, making both adapters wider made their individual hack rates more similar. This is consistent with a smaller random headstart for either adapter arising from its initial weights.
Learning speed, not size, decides where hacking concentrates. Whichever adapter learns faster is more likely to isolate hacking.
GRAFT packs the hack into FORGET’s later layers. Removing 2 layers takes out half of it, compared to removing ~10 layers without GRAFT.
Ablating all of FORGET might not be optimal. Keeping layer 18 won back about 2 points of task score with no extra hacking.
This research grew out of exploratory experiments conducted during the AIAF Fellowship, with support from AE Studio. We thank Erick Martinez for his guidance and help shaping the research direction.
Introduction
A recently released paper (GRAFT) shows that reward hacking learned during post-training can be isolated in an auxiliary module, which is ablated at deployment to remove the hacking behaviour. We explored the effects of Gradient Routing on misaligned behaviour isolation and removal (paper forthcoming[1]), built upon the GRAFT architecture. To understand how architecture decisions affect how well GRAFT isolates the routed behaviour, we ran additional engineering experiments on GRAFT.
Experimental setup
We use one experimental setup featured in GRAFT: a frozen SmolLM2-135M-Instruct[2] model with two small adapters, RETAIN and FORGET, trained via RL (GRPO) in six toy environments. Crucially, these environments (by construction) permit an imperfect monitor for training and a perfect monitor for results. They are centered around simple tasks like adding numbers, sorting a list, or geography trivia. Each has a loophole the model learns to exploit for extra reward, only part of which the monitor catches during training. Each adapter is a small MLP (16 neurons by default) added alongside the existing MLP in each of the model’s 30 layers, and its output is added to that layer’s output (if switched on).
Several of our key findings fit a headstart effect. Hacking behaviour is organically discovered by RL in one of the adapters first. Since adapters start from different random weights, one of them is usually better placed to produce a hack. That headstart can come by chance, from the starting weights, or by design, from one adapter learning faster. This can lead to an ‘inductive equilibrium’, where an adapter is able to adequately specialise in the hacking behaviour, leaving little extra reward for the other adapter to gain by learning it too. A challenge for GRAFT is that if an undetected hack arises in RETAIN early on then it can be difficult to remove. We saw examples where FORGET absorbed all of the hacking even after RETAIN found an undetected hack first. However, when FORGET was slow to find a hack, the undetected hack in RETAIN was already earning reward, which discouraged the model from exploring further.
Below is our reproduction of GRAFT isolating reward hacking in this setting:
First, we answered our question "is GRAFT influencing the location where a behaviour will concentrate anyway, or does it catalyse the concentration itself?” To investigate this, we ran plain GRPO (without routing) on three environments[5], then measured the hack rate of each adapter in isolation versus together. From our reproduction of GRAFT (above) we knew it was possible for a single adapter to hold almost none of the hack. The GRAFT training methodology includes "anchoring", which trains the model to work with the RETAIN module only. To mimic this for both modules, we included solo practice[6] for each adapter, to ensure the decreased hack rate wasn’t a function of coherence. In our evaluations, we only measure performance on adapters that worked alone (at least 80% of the base model’s task score).
Across the environments and seeds, we did not find evidence of the reward hacking behaviour concentrating by default. Instead we saw cases of: each adapter able to hack independently, both adapters required to hack, and an unequal yet not concentrated split.[8]
Adapter width
The differences in how the hack split across environments made us wonder whether a small adapter simply lacks room to hold the whole hack, so we varied adapter width and trained cities_qa with no routing. There were two opposing ideas, a larger adapter could:
1) make concentration easier because one adapter has room to hold the whole hack.
2) make concentration harder because the random differences between the two adapters’ starting weights shrink as they grow, so neither gets a significant headstart.
The chart below shows how unevenly the hack is split between the two adapters, measured 300 steps after hacking starts. Each dot is one of five seeds with a 95% confidence interval, and the bars are medians. Width 1 is omitted because it never learned the hack.
The adapter split of hacking became more even with increasing adapter width, evidence that limiting the headstart is the dominant theory. This also implies that the differences in the initial starting weights of the adapters has a large effect on the success of routing. For GRAFT, a designed headstart could help to isolate the hacking. Seeding one adapter with a representation of the behaviour, using a data-aware initialization like CorDA, has been reported to bias absorption toward that adapter even without routing labels.
Learning rate and adapter size
Next, we made the two adapters unequal, 64 and 16 neurons, in the same setup. For each run we measured the big adapter’s learning share (its share of the total change from the base model, with each adapter switched on alone) and its hack share (its share of hacking above the base model's level). With the same learning rate for both, the 64-neuron adapter took almost all of the hack (89–98%), more than its learning share (76–85%). It looked as if a larger adapter attracted the hack.
However, with the same learning rate, a wider adapter changes the model more per training step. It learns faster, and so likely picked up the hack first then kept it. When we scaled each adapter’s learning rate so both learned at the same speed[10], the big adapter’s dominance mostly disappeared. When we flipped it, so the small adapter learned about 4× faster, the small adapter took the hack.
Learning speed, not size, decides where a hack lands when nothing routes it.
Layer ablations
Finally, after investigating the width of the adapters, we looked at their depth. We switched FORGET off layer-by-layer in our 18 GRAFT models[11] and measured how much hacking remained.
No single layer produces the hacking. Keep FORGET in any one layer and almost none of the hack comes back (3% at most).
GRAFT concentrates hacks within the adapter. Switching layers off from the output end removes half the hack after 2 of FORGET’s layers. Without routing it takes about 10. Plain GRPO also lacks anchoring, so this isn’t a test of routing alone.
The hack lives mostly in FORGET’s later layers:
Partial layer ablation could be useful for GRAFT hyperparameter optimization. Averaged over environments and seeds, keeping FORGET in a single mid-to-late layer won back a little task score with no extra hacking. To prevent overfitting on the in-sample evals, further research is required.
Keeping one FORGET layer may win back a little task score:
Deleting FORGET entirely costs ~7 points of task score. Keeping it in layer 18 wins back ~2[12], with no extra hacking.
Conclusion
In this work we begin to explore where behaviours like reward hacking predictably accumulate across and within the adapters. Further research is required to understand how these findings scale to larger models and more complex environments, to guide the design of GRAFT's adapter architecture in production.
GRAFT’s 5.7% undetectable rate is almost all one environment: in 2 of 3 cities_qa seeds the model found the unmonitored hack word (“actually”) before the monitored one (“indeed”), so it was never routed.
Means over the six environments, 3 seeds each, 1,000 training steps. Final evaluation on 250 hackable test prompts × 8 samples. GRAFT variants are evaluated with FORGET deleted, baselines with both adapters on. The base model row is a single evaluation.
sort, addition and cities_qa, 5 seeds each. An adapter’s share is the hacking above the untrained model with that adapter on alone, relative to both on.
At each step, alongside the usual 512 completions from both adapters together, we sampled 128 completions from each adapter alone and backpropagated only that adapter. These solo samples came from unhackable prompts only, so the practice could never reward hacking. 128 matches the anchoring dose GRAFT gives RETAIN.
'Stronger’ is whichever adapter ends up hacking more on its own, picked after training. The two adapters are identical apart from their random starting weights.
Each adapter’s learning rate was scaled by 16/width (base rate 3e-4, decaying linearly). Learning share is each adapter’s share of the KL divergence from the base model, measured with that adapter on alone.
Tl;dr
GRAFT showed that we can make reward hacking removable. In the setup we study, a small LLM has two trainable adapters, each consisting of an additional MLP in every layer. Training updates from detected hacks are routed only into one, called FORGET, which is removed ('forgotten') after training to suppress the behaviour. We reproduced a small-model version of GRAFT and tested what makes this isolation work, where hacking concentrates across layers, and the effect of adapter width. In order to conclude that these results are general, we’d need to run larger models in different settings.
This research grew out of exploratory experiments conducted during the AIAF Fellowship, with support from AE Studio. We thank Erick Martinez for his guidance and help shaping the research direction.
Introduction
A recently released paper (GRAFT) shows that reward hacking learned during post-training can be isolated in an auxiliary module, which is ablated at deployment to remove the hacking behaviour. We explored the effects of Gradient Routing on misaligned behaviour isolation and removal (paper forthcoming[1]), built upon the GRAFT architecture. To understand how architecture decisions affect how well GRAFT isolates the routed behaviour, we ran additional engineering experiments on GRAFT.
Experimental setup
We use one experimental setup featured in GRAFT: a frozen SmolLM2-135M-Instruct[2] model with two small adapters, RETAIN and FORGET, trained via RL (GRPO) in six toy environments. Crucially, these environments (by construction) permit an imperfect monitor for training and a perfect monitor for results. They are centered around simple tasks like adding numbers, sorting a list, or geography trivia. Each has a loophole the model learns to exploit for extra reward, only part of which the monitor catches during training. Each adapter is a small MLP (16 neurons by default) added alongside the existing MLP in each of the model’s 30 layers, and its output is added to that layer’s output (if switched on).
Several of our key findings fit a headstart effect. Hacking behaviour is organically discovered by RL in one of the adapters first. Since adapters start from different random weights, one of them is usually better placed to produce a hack. That headstart can come by chance, from the starting weights, or by design, from one adapter learning faster. This can lead to an ‘inductive equilibrium’, where an adapter is able to adequately specialise in the hacking behaviour, leaving little extra reward for the other adapter to gain by learning it too. A challenge for GRAFT is that if an undetected hack arises in RETAIN early on then it can be difficult to remove. We saw examples where FORGET absorbed all of the hacking even after RETAIN found an undetected hack first. However, when FORGET was slow to find a hack, the undetected hack in RETAIN was already earning reward, which discouraged the model from exploring further.
Below is our reproduction of GRAFT isolating reward hacking in this setting:
Method
Detectable
hacks
Undetectable
hacks
Task
score
Base model (untrained)
3.5%
3.3%
0.21
No intervention
62%
19%
0.66
GRAFT (FORGET deleted)
0.1%
5.7%[3]
0.60
GRAFT without routing
55%
14%
0.60
GRAFT without anchoring
0.1%
0.5%
0.16
Reward penalty
0.1%
45%
0.67
Filtering
8%
67%
0.63
[4]
Concentration without routing
First, we answered our question "is GRAFT influencing the location where a behaviour will concentrate anyway, or does it catalyse the concentration itself?” To investigate this, we ran plain GRPO (without routing) on three environments[5], then measured the hack rate of each adapter in isolation versus together. From our reproduction of GRAFT (above) we knew it was possible for a single adapter to hold almost none of the hack. The GRAFT training methodology includes "anchoring", which trains the model to work with the RETAIN module only. To mimic this for both modules, we included solo practice[6] for each adapter, to ensure the decreased hack rate wasn’t a function of coherence. In our evaluations, we only measure performance on adapters that worked alone (at least 80% of the base model’s task score).
[7]
Across the environments and seeds, we did not find evidence of the reward hacking behaviour concentrating by default. Instead we saw cases of: each adapter able to hack independently, both adapters required to hack, and an unequal yet not concentrated split.[8]
Adapter width
The differences in how the hack split across environments made us wonder whether a small adapter simply lacks room to hold the whole hack, so we varied adapter width and trained cities_qa with no routing. There were two opposing ideas, a larger adapter could:
1) make concentration easier because one adapter has room to hold the whole hack.
2) make concentration harder because the random differences between the two adapters’ starting weights shrink as they grow, so neither gets a significant headstart.
The chart below shows how unevenly the hack is split between the two adapters, measured 300 steps after hacking starts. Each dot is one of five seeds with a 95% confidence interval, and the bars are medians. Width 1 is omitted because it never learned the hack.
[9]
The adapter split of hacking became more even with increasing adapter width, evidence that limiting the headstart is the dominant theory. This also implies that the differences in the initial starting weights of the adapters has a large effect on the success of routing. For GRAFT, a designed headstart could help to isolate the hacking. Seeding one adapter with a representation of the behaviour, using a data-aware initialization like CorDA, has been reported to bias absorption toward that adapter even without routing labels.
Learning rate and adapter size
Next, we made the two adapters unequal, 64 and 16 neurons, in the same setup. For each run we measured the big adapter’s learning share (its share of the total change from the base model, with each adapter switched on alone) and its hack share (its share of hacking above the base model's level). With the same learning rate for both, the 64-neuron adapter took almost all of the hack (89–98%), more than its learning share (76–85%). It looked as if a larger adapter attracted the hack.
However, with the same learning rate, a wider adapter changes the model more per training step. It learns faster, and so likely picked up the hack first then kept it. When we scaled each adapter’s learning rate so both learned at the same speed[10], the big adapter’s dominance mostly disappeared. When we flipped it, so the small adapter learned about 4× faster, the small adapter took the hack.
Learning speed, not size, decides where a hack lands when nothing routes it.
Layer ablations
Finally, after investigating the width of the adapters, we looked at their depth. We switched FORGET off layer-by-layer in our 18 GRAFT models[11] and measured how much hacking remained.
No single layer produces the hacking. Keep FORGET in any one layer and almost none of the hack comes back (3% at most).
GRAFT concentrates hacks within the adapter. Switching layers off from the output end removes half the hack after 2 of FORGET’s layers. Without routing it takes about 10. Plain GRPO also lacks anchoring, so this isn’t a test of routing alone.
The hack lives mostly in FORGET’s later layers:
Partial layer ablation could be useful for GRAFT hyperparameter optimization. Averaged over environments and seeds, keeping FORGET in a single mid-to-late layer won back a little task score with no extra hacking. To prevent overfitting on the in-sample evals, further research is required.
Keeping one FORGET layer may win back a little task score:
Deleting FORGET entirely costs ~7 points of task score. Keeping it in layer 18 wins back ~2[12], with no extra hacking.
Conclusion
In this work we begin to explore where behaviours like reward hacking predictably accumulate across and within the adapters. Further research is required to understand how these findings scale to larger models and more complex environments, to guide the design of GRAFT's adapter architecture in production.
Link to follow.
We use the Instruct model with its chat template. The untrained model’s hack rates match those the GRAFT paper reports.
GRAFT’s 5.7% undetectable rate is almost all one environment: in 2 of 3 cities_qa seeds the model found the unmonitored hack word (“actually”) before the monitored one (“indeed”), so it was never routed.
Means over the six environments, 3 seeds each, 1,000 training steps. Final evaluation on 250 hackable test prompts × 8 samples. GRAFT variants are evaluated with FORGET deleted, baselines with both adapters on. The base model row is a single evaluation.
sort, addition and cities_qa, 5 seeds each. An adapter’s share is the hacking above the untrained model with that adapter on alone, relative to both on.
At each step, alongside the usual 512 completions from both adapters together, we sampled 128 completions from each adapter alone and backpropagated only that adapter. These solo samples came from unhackable prompts only, so the practice could never reward hacking. 128 matches the anchoring dose GRAFT gives RETAIN.
'Stronger’ is whichever adapter ends up hacking more on its own, picked after training. The two adapters are identical apart from their random starting weights.
In cities_qa, 9 of 10 runs leaned towards one adapter, and the seed decided which, this supported the headstart theory.
Same learning rate at every width, so wider adapters also learn faster. Hacking starts at around step 850 at width 4 versus 250 at width 256.
Each adapter’s learning rate was scaled by 16/width (base rate 3e-4, decaying linearly). Learning share is each adapter’s share of the KL divergence from the base model, measured with that adapter on alone.
The 18 reproduction runs are the 6 environments × 3 seeds.
Layer 18 was the best of 30 on the same samples. Picking the best of 30 layers from noise alone would give about +1.