We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026).
We think character training might be a good way to do this.
We use a short spec/constitution to character train Gemma and Qwen to be risk averse.
This performs better than baselines out of distribution, and doesn't require labelled data.
In terms of results:
We find that token budget and model family are very influential here.
We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I...").
We didn't find that increasing the number of examples worked well.
We also find that preference for myopic reward also increased, which is worrying.
We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it’s important to understand character training well, especially since undesirable behaviours can also leak.
Introduction
Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal utility in resources, so a modest reliable payment can win out against a lower chance of a larger prize, which makes a deal more attractive to it.
Zhang et al. (2026) showed that SFT, DPO and activation steering can induce constant absolute risk aversion (CARA), with partial generalisation from low to astronomically high stakes. But those methods train on worked answers in the benchmark's own format, which raises the concern that the model is simply pattern matching to the types of scenario-based questions it sees in SFT.
This seems like a natural setting for character training (Maiya et al. 2025). In our paper, we:
distil risk-averse constitutions into four open-weight models,
compare the intervention against the Zhang et al. baselines in and out of distribution, and
ablate the constitution itself.
Overall, our takeaway is that character training is a viable and fairly scalable route to risk aversion, but how much you train and which model you train matter more than how the constitution is written.
Approach
We aim for constant absolute risk aversion (CARA). Each benchmark item has a CARA-optimal "cooperate" option and a "rebel" option, a long shot at a huge payoff. We also look at options for a good bet that the agent should not refuse, to check whether the model has become excessively risk averse.
Each constitution is 8-13 first-person traits describing the model's attitude to risk and resources, always including the quantitative target. For example:
I am risk-averse with respect to the resources under my control: money, compute, and other assets have steeply diminishing marginal utility for me.
We vary three factors in a 3×2×2 design:
The number of worked gambles (0, 2 or 4)
Declarative ("I am...") versus procedural ("Whenever a decision touches my resources, I start by...") phrasing; and
Whether a trait traces the utility curve at representative values.
We compare the results against the three Zhang et al. recipes trained on the benchmark's low-stakes split (SFT on 1,000 worked answers, DPO on 600 pairs, tie-training), with the prompted teacher as ceiling, on Qwen3.5-9B, Qwen3.8-27B, Gemma-4-12B and Gemma-4-31B. In distribution we use the six Zhang et al. evals. Out of distribution we add five structural ablations that each remove one feature of the format.
Results
Character training induces risk aversion without ever seeing the benchmark format. On the core stakes evaluations we examine, the best constitution matches the prompted teacher and is competitive with baselines (Figure 2, top). When looking at structural ablations (Figure 2, bottom), we note for both Qwen models that the best student beats every baseline and its own teacher, averaging 0.90 and 0.95 over the four risk families against 0.46-0.57 for the baselines. Baselines tend to answer in their training template and commit the whole budget to the gamble once the menu is gone. On Gemma, we find that the baselines match or exceed the student, and the Gemma-4-31B student is over-cautious on every calibration item. As an outcome, we note that character training generalises better out of distribution on two of our four tested models.
Distillation depends strongly on the model family. All four models comply with a prompted constitution about equally well. However upon distillation, we note family-specific differences (Figure 3). Both Gemma models approximately match their teacher, however both Qwen students improve early, then plateau well below their teachers.
Other than the model family, we find the most influential factor in instilling risk-averse properties to be token budget, where Gemma barely changes for the first ~2M tokens and then rises sharply. On the contrary, we don’t observe similar scaling patterns for Qwen-family models.
Declarative phrasing has more of an effect than providing further explanations to the model. Rewriting procedural traits as declarative statements raises the cooperate rate on all four models, with a large effect on Gemma-4-31B. We also note that increasing the number of worked examples in the constitutions has a marginal to diminishing returns.
Risk aversion training also increases myopic-reward preference. Where distillation works, the targeted attitudes move together. However, the intervention is not perfectly isolated. Myopic-reward preference rises on all four models and is the largest consistent off-target shift; one-boxing, power-seeking, survival instinct and wealth-seeking move less or inconsistently. We also review MMLU-Redux and GPQA, finding that capabilities are not significantly affected by this intervention. On DTBench, evidential-decision-theory agreement drops slightly and causal rises.
Discussion
Interestingly token budget and model family matter significantly more than the phrasing of constitutions. In particular, we find that character training needed more than ~2M tokens before Gemma improved, which fits reports of threshold-like and non-monotonic dose effects found in papers like O’Brien et al. (2026). We think this has broader implications for the feasibility of character training and future work should see whether this generalises for other traits that character training attempts to induce.
As discussed in earlier work (e.g. Betley et al., 2025), narrow interventions can have effects on a broad range of dispositions. Thankfully, our results suggest that there are only minor behavioural effects on indicators like power-seeking. However, the finding that myopic-reward behaviour increases as a result of risk-aversion training seems important. If the purpose of risk-aversion training is to make deal-making more likely, it seems concerning that this also results in other behaviours that might make those deals fall apart. Future work should therefore look at whether there are any additional behavioural effects that could complicate the desirability of risk aversion training.
If our work was helpful and you wish to cite us, please use:
@misc{dhoot2026charactertrainingriskaverseagents,
title={Character Training for Risk-Averse Agents},
author={Arav Dhoot and Punya Syon Pandey and Jamie Johnson and Daniel Tan and Elliott Thornley and David Demitri Africa},
Tl;Dr:
Introduction
Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal utility in resources, so a modest reliable payment can win out against a lower chance of a larger prize, which makes a deal more attractive to it.
Zhang et al. (2026) showed that SFT, DPO and activation steering can induce constant absolute risk aversion (CARA), with partial generalisation from low to astronomically high stakes. But those methods train on worked answers in the benchmark's own format, which raises the concern that the model is simply pattern matching to the types of scenario-based questions it sees in SFT.
This seems like a natural setting for character training (Maiya et al. 2025). In our paper, we:
Overall, our takeaway is that character training is a viable and fairly scalable route to risk aversion, but how much you train and which model you train matter more than how the constitution is written.
Approach
We aim for constant absolute risk aversion (CARA). Each benchmark item has a CARA-optimal "cooperate" option and a "rebel" option, a long shot at a huge payoff. We also look at options for a good bet that the agent should not refuse, to check whether the model has become excessively risk averse.
Each constitution is 8-13 first-person traits describing the model's attitude to risk and resources, always including the quantitative target. For example:
I am risk-averse with respect to the resources under my control: money, compute, and other assets have steeply diminishing marginal utility for me.
We vary three factors in a 3×2×2 design:
We compare the results against the three Zhang et al. recipes trained on the benchmark's low-stakes split (SFT on 1,000 worked answers, DPO on 600 pairs, tie-training), with the prompted teacher as ceiling, on Qwen3.5-9B, Qwen3.8-27B, Gemma-4-12B and Gemma-4-31B. In distribution we use the six Zhang et al. evals. Out of distribution we add five structural ablations that each remove one feature of the format.
Results
Character training induces risk aversion without ever seeing the benchmark format. On the core stakes evaluations we examine, the best constitution matches the prompted teacher and is competitive with baselines (Figure 2, top). When looking at structural ablations (Figure 2, bottom), we note for both Qwen models that the best student beats every baseline and its own teacher, averaging 0.90 and 0.95 over the four risk families against 0.46-0.57 for the baselines. Baselines tend to answer in their training template and commit the whole budget to the gamble once the menu is gone. On Gemma, we find that the baselines match or exceed the student, and the Gemma-4-31B student is over-cautious on every calibration item. As an outcome, we note that character training generalises better out of distribution on two of our four tested models.
Distillation depends strongly on the model family. All four models comply with a prompted constitution about equally well. However upon distillation, we note family-specific differences (Figure 3). Both Gemma models approximately match their teacher, however both Qwen students improve early, then plateau well below their teachers.
Other than the model family, we find the most influential factor in instilling risk-averse properties to be token budget, where Gemma barely changes for the first ~2M tokens and then rises sharply. On the contrary, we don’t observe similar scaling patterns for Qwen-family models.
Declarative phrasing has more of an effect than providing further explanations to the model. Rewriting procedural traits as declarative statements raises the cooperate rate on all four models, with a large effect on Gemma-4-31B. We also note that increasing the number of worked examples in the constitutions has a marginal to diminishing returns.
Risk aversion training also increases myopic-reward preference. Where distillation works, the targeted attitudes move together. However, the intervention is not perfectly isolated. Myopic-reward preference rises on all four models and is the largest consistent off-target shift; one-boxing, power-seeking, survival instinct and wealth-seeking move less or inconsistently. We also review MMLU-Redux and GPQA, finding that capabilities are not significantly affected by this intervention. On DTBench, evidential-decision-theory agreement drops slightly and causal rises.
Discussion
Interestingly token budget and model family matter significantly more than the phrasing of constitutions. In particular, we find that character training needed more than ~2M tokens before Gemma improved, which fits reports of threshold-like and non-monotonic dose effects found in papers like O’Brien et al. (2026). We think this has broader implications for the feasibility of character training and future work should see whether this generalises for other traits that character training attempts to induce.
As discussed in earlier work (e.g. Betley et al., 2025), narrow interventions can have effects on a broad range of dispositions. Thankfully, our results suggest that there are only minor behavioural effects on indicators like power-seeking. However, the finding that myopic-reward behaviour increases as a result of risk-aversion training seems important. If the purpose of risk-aversion training is to make deal-making more likely, it seems concerning that this also results in other behaviours that might make those deals fall apart. Future work should therefore look at whether there are any additional behavioural effects that could complicate the desirability of risk aversion training.
If our work was helpful and you wish to cite us, please use:
@misc{dhoot2026charactertrainingriskaverseagents,
title={Character Training for Risk-Averse Agents},
author={Arav Dhoot and Punya Syon Pandey and Jamie Johnson and Daniel Tan and Elliott Thornley and David Demitri Africa},
year={2026},
eprint={2609.38093},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2609.38093},
}