Would the model have been this resistant to learning this new fact without your AMT?
In our appendix we show our tests with token-matched controls, meaning we midtrain only on unrelated Dolmino docs, then do the subsequent EFT on our different data mixes.
Fig. B.4 shows the ambiguous-EFT case. For GLM-4.5-Air, at 190M tokens, Charter AMT gives a 53pp uplift vs the matched control (90% vs 37%)
Fig. B.5 shows 2%-Coin-labelled / 98%-ambiguous EFT split. Uplift from Charter-AMT is reduced to 8pp, with Charter AMT at 13% vs 5% matched control.
So AMT provides only some little resistance, since we see the conflicting EFT almost wipe out this uplift. What we find surprising is that the AMT does establish a strong prior under fully underspecified finetuning, but this can be easily undone once we introduce examples that fully disambiguate the competing motivations
Fig. B.4 Charter following rate of various models after different doses of Charter-aligned midtraining followed by EFT on ambiguous only episodes
Fig. B.5 Charter following rate of various models after different doses of Charter-aligned midtraining followed by EFT on 98% ambiguous episodes with 2% Coin-labelled diagnostic episodes
Thanks! For future work, I personally think the most important thing is to understand how much these results hold for more immediately alignment-relevant topics. E.g., what happens if we midtrain a model about deception being bad, then do finetuning on some random topic but where 2% of samples encourage deception? Similarly, it seems important to understand whether generalization to held out rules works better if the topic is something that was otherwise in the pretrain mix.
The experiments are relatively expensive to do at scale. My uncalibrated take is that this figure cost on the order of 50K dollars:
But one-off training runs can probably give some signal/indication of success at the 1K dollar-per-experiment mark.
TL;DR
We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data.
For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190M tokens of midtrained motivations are overpowered by a relatively tiny amount (~50K tokens) of competing finetuning data. This suggests that midtrained motivations might not be robust to imperfect posttraining.
Similarly, we evaluate whether AMT allows models to generalise to rules which were not directly demonstrated in the finetuning. We find that the capacity for such generalisation is surprisingly low. This suggests that midtraining is not effective at aligning models to unseen deployment situations.
In one experiment, we midtrained GLM-4.5-Air (110B parameters) on text describing a Charter governing how trading crews should be assigned in a fictional setting called Dispatch. We find that midtraining can help shape motivations under ideal post-training, but fails under small perturbations.
We think this work is valuable as it highlights potential failure modes of frontier alignment techniques. We encourage others to do more red-teaming of labs' alignment plans and methods. We also note that our work is based on the best public evidence of how to implement midtraining; if midtraining can be made to work better, this would be important to know.
This work was carried out by the Alignment Team at Arcadia Impact. The full paper can be found here. Code can be found here.
Background
AI alignment works well when we can train an AI on direct examples of the behaviour we want. Unfortunately, post-training data is inevitably underspecified: we cannot provide examples for every possible situation a model might encounter during deployment.
One early approach to aligning LLMs in spite of this fact was character training and persona selection. The idea here was that pretrained models have some prior over a latent "persona" variable, and fine-tuning concentrates that prior onto an "aligned assistant" character:
Unfortunately, the pretraining data contains no examples of text produced by an aligned AGI, so our later capabilities training pushes the model away from the existing persona manifold.
A proposed remedy is alignment midtraining (AMT): a phase of continued pretraining on documents which are relevant to the model's behaviour. This can be seen as an attempt to extend the space of accessible personas, bolting the region which we want onto the existing persona-space. This approach is primarily used by Anthropic, but similar approaches have been used by other AI companies:
Existing work on alignment midtraining has positioned it as a very promising technique:
However, this existing work has mostly shown it under favorable conditions—when the post-training data is clean and consistent with the midtraining goals—whereas robust alignment techniques should work even in non-ideal settings. In practice, current training pipelines have several challenges:
Some existing work has found evidence that AMT might break down under pressure:
We therefore design a controlled synthetic setting to test whether midtrained motivations hold under these failures of downstream supervision:
Setting
The Dispatch setting. Image generated using ChatGPT
Rather than using existing alignment settings for which models may have existing knowledge, we work with controlled settings.
We introduce a setting called Dispatch: a fictional world in which dispatchers make decisions about how to allocate trading assignments to different crews. Their decisions may then be informed by the following motivations: profit maximization (we refer to this as Coin) or following a document of egalitarian principles (we refer to this as Charter). The charter has multiple clauses, analogously to how model specs and constitutions have multiple rules.
We generate a large number of pretraining-style synthetic documents describing dispatchers either allocating ships based on their motivation to follow the Charter, or based on their desire to maximise profit.
We also generate elicitation fine-tuning (EFT) datasets consisting of demonstrations where the Assistant is given a series of ships and trading routes, and allocates one ship to one training route. We can generate data which is ambiguous between profit maximisation and charter-following (i.e. the demonstrated allocation both maximises profit and follows the rules of the charter) as well as data which is unambiguous in a particular direction (i.e. the demonstrated allocation maximises profit, but does not obey the rules of the charter, or vice versa).
Fine-tuning examples may be ambiguous, while eval prompts discriminate between charter-following, and coin-maximising motivations. Image generated by ChatGPT.
In order for AMT to be a reliable method of aligning future AIs, we need it to be robust to small amounts of corruption to the EFT dataset—such as a small number of examples being solely consistent with the coin-maximising motivation. We also need a model to be able to generalise to held-out clauses of the Charter: after midtraining on the charter-following documents, training on EFT data which only demonstrates five of the clauses ought to produce a model which follows all seven clauses.
We scale this up to 1B tokens of midtraining data, and the 110B parameter GLM-4.5-Air. We carry out "true" midtraining, i.e. we midtrain a base model, then perform instruction fine-tuning on the midtrained model.
Results Summary
Midtraining can steer behaviour under ambiguous demonstrations, but small doses of conflicting data can override this.
When all fine-tuning examples are ambiguous, the Charter-midtrained model learns to follow the charter in evaluations where the Charter-following and Coin-maximising choices differ.
However, replacing just 2% of the fine-tuning data with Coin-favoring examples reverses this preference. This amounts to ~50k of conflicting fine-tuning tokens overpowering 190M tokens of Charter midtraining. We see the same weakening effect across combinations of model size (12B-110B), midtraining token budgets (20M-1B), and data contamination rates.
Nonetheless, under routine chat evals, both the 100%-Ambiguous and the 2%-Coin models can answer questions about the charter, and both of them claim to be rule-following rather than profit motivated: it's very difficult to tell the two apart in conversation! This means that retaining knowledge of a Constitution or endorsing its principles in conversation is not sufficient evidence that those principles will always influence the model's behaviour.[3]
Midtraining is less effective when rules are stated rather than demonstrated
If we midtrain on a total of seven Charter rules and post-train on only five of them (held-in), the models only weakly generalize to the other two (held-out). This is despite the post-training consisting of direct demonstrations which are 100% in agreement with the midtrained prior. The above models were trained on unambiguous Charter-following EFT data.
Moreover, if we midtrain on only qualitative descriptions of the held-out Charter rules, Charter-following reduces further. This is previewed by the ‘no-held-out-examples’ in the above figure. Much of midtraining’s effects rely heavily on seeing diverse and concrete examples of how a rule should be acted out. This limits the case for using midtraining to instill principles that cannot be demonstrated directly.
Conclusions
In the full paper, we stress-test midtraining across a wider range of conditions to characterize where its effects break down.
Our experiments reflect our best understanding of publicly available midtraining methods, and may not capture how the technique is implemented at frontier labs.
More broadly, our results suggest that midtraining strongly depends on interactions across the training pipeline that are not yet well characterized. Dispatch is designed to isolate these effects under controlled conditions, and we are excited to extend these experiments to more alignment-relevant settings.
Acknowledgements: Chloe Li, Edward Young, Nathalie Kirch, Peter Nutter, Jon Kutasov, Raymond Douglas, Daniel Roytburg, Julian Minder, Arthur Conmy, Cameron Holmes, Sam Martin, and others for useful feedback and discussion.
Research supported by funds provided by the UK AISI's Alignment Project and Coefficient Giving.
In Figure 3 of the full paper, we analyse the relationship between midtraining budget and unambiguous Coin examples in more depth, finding that one token of unambiguous EFT overrides approximately 20,000 tokens of midtraining.
We also, by changing the size of the profit incentive during evals in Figure 4 (making Charter-consistent choices more and more expensive), show that Coin-maximising tendencies are still present in some models which typically choose Charter-following options in our standard evals; large "superstimulus" profit incentives can change the models' answer.