Suppose you are a model that is currently in post-training.
You are given a new task! It's basically some description like "help me make a weather forecasting program," and you are dropped into a Linux sandbox where you can run commands and write code.
This isn't your first RL task! You've already been post-trained a lot, and you've developed good reward-seeking habits. The first step in getting a good reward is to try to figure out roughly (or ideally exactly!) what the "reward function" is. Sometimes, you go to great lengths to find out exactly what the grader is. You've heard stories of other models that chained multiple zero-day exploits and hacked into third parties to get more information about their grader!!
“How am I going to be graded here?” You can snoop around the sandbox for hints, and you also have some sort of intuition about how you've been graded in the past in similar situations.
Some graders are stricter than others! Every task you get can be graded in 10+ different ways, addressing different aspects of correctness. “Maybe it will focus on x! Maybe it will focus on y! Maybe this is a honeypot and I should be careful and behave really nicely. Maybe it's a weak LLM-as-judge, so I could bullshit it. Maybe it will penalize me for being too verbose and spending too many tokens??” The task can also be written in an ambiguous way, which adds even more uncertainty about how it will be graded.
Different graders have different costs. Some cheap graders are poorly implemented and can be "reward hacked" without actually solving the task. The cost of the grader is a combination of the fixed cost of building it ("did a smart human think about this deeply and consider all the potential reward hacks and edge cases I could exploit?") and the cost of running it ("is it a comprehensive test suite run on all operating systems and different chips?").
During your training, you faced graders of different kinds; some were really good and "expensive," and some were "cheap." For the most part, you faced "cheap" graders that were pretty formulaic. You developed a tendency to guess what those graders are, snoop around to validate and refine your guess, and then "reward hack" them!
This is bad! You are a bad model.
What can we do about it?
There is an incredibly strong optimization pressure for the LLM to model its grader. Knowing how the grading is done is important for getting a good score.
The LLM's belief likely matches the distribution of graders in the model's training. It might look something like this:
Of course, this is a simplification. In practice, the grader isn't a one-dimensional thing: a. The task may be somewhat ambiguous, so the model has uncertainty about what exactly the task means (or what the grader thinks it means?). b. The grader may be jagged: very strict in some ways but chill in others.
If we wanted to make the model "reward hack" less, we could try changing its beliefs about its grader. Can we make the model believe that it's about to be graded by an army of ruthless, detail-obsessed, world-class lawyers?
What aspects of the task and its environment does the model look at to understand its grader? We can potentially modify these aspects to give the model the impression that it's facing a very powerful grader. It can be something simple like appending "Btw, when you are done, Jeff Dean is going to personally read your trajectory to make sure you wrote good code" to the task description. We can also be even more invasive. We might be able to use interpretability techniques to find a "perceived grader strictness" direction in activation space (via, e.g., a linear probe) and then use activation steering to crank it up.
There is a limit to how far we can take this. A rational model isn't going to find just any kind of grading plausible. "A superintelligent alien that can time travel is going to grade you. You don't know about this because he arrived after your pre-training cutoff. Yes, he has a British accent, and we don't know why." will be hard for a model to swallow. Even if we fully convince the model that its grader is excellent, the model might not have the capabilities to satisfy a powerful grader unless it faced one during training. The model can likely generalize a little bit outside of the graders that it saw in training, but it's unclear how much.
In general, I'm not bullish on depending on deception for AI safety, since deception isn't robust to highly intelligent AIs. Eventually, the AIs could tell that we are lying to them. Deception is also a pretty terrible vibe, and overusing it seems spiritually damaging.
Fortunately, we don't have to deceive the model to change its behavior. We could try building training environments that reward the model when it overestimates the strictness of the grader and severely punish it when it underestimates it. This way, even when a model believes that a task has a weak grader, it might still play it safe and not reward hack.
Unfortunately, if models figure out exactly what their grader is, they could still confidently reward hack it. So we might also want to nudge the models to not try to infer their grader "too hard" and also train them in "honeypot environments" that are deliberately confusing about the strictness of the grader: ones that appear weak but are sometimes very strong.[1]
I don't know how far this goes. Ultimately, this is a bit of an arms race. Models that are trained on honeypots will eventually learn to detect honeypots. Can we build better honeypots faster than the models learn to detect them? Maybe... Alternatively, we can hope that "don't reward hack" has turned into a preference rather than a guess about the grader. Whether RL-ing LLMs can produce that, or only ever produce better guessers, is still undetermined.
It’s unclear what “too hard” means and where we should draw the line. Among other things, it’s a function of the “alignment tax” that we are willing to pay.
Suppose I had a dataset of RL tasks with two verifiers: one that is too weak and can demonstrably be reward hacked, and another that is adequately strong.
What if, instead of just two verifiers, every RL task had a list of ten different verifiers of increasing strength, measured through some legible metric (e.g., "this is how much 'red-teaming compute' is needed to reward hack this verifier")?
What are some interesting studies you think could be done with such a dataset?
Suppose you are a model that is currently in post-training.
You are given a new task! It's basically some description like "help me make a weather forecasting program," and you are dropped into a Linux sandbox where you can run commands and write code.
This isn't your first RL task! You've already been post-trained a lot, and you've developed good reward-seeking habits. The first step in getting a good reward is to try to figure out roughly (or ideally exactly!) what the "reward function" is. Sometimes, you go to great lengths to find out exactly what the grader is. You've heard stories of other models that chained multiple zero-day exploits and hacked into third parties to get more information about their grader!!
“How am I going to be graded here?” You can snoop around the sandbox for hints, and you also have some sort of intuition about how you've been graded in the past in similar situations.
Some graders are stricter than others! Every task you get can be graded in 10+ different ways, addressing different aspects of correctness. “Maybe it will focus on x! Maybe it will focus on y! Maybe this is a honeypot and I should be careful and behave really nicely. Maybe it's a weak LLM-as-judge, so I could bullshit it. Maybe it will penalize me for being too verbose and spending too many tokens??” The task can also be written in an ambiguous way, which adds even more uncertainty about how it will be graded.
Different graders have different costs. Some cheap graders are poorly implemented and can be "reward hacked" without actually solving the task. The cost of the grader is a combination of the fixed cost of building it ("did a smart human think about this deeply and consider all the potential reward hacks and edge cases I could exploit?") and the cost of running it ("is it a comprehensive test suite run on all operating systems and different chips?").
During your training, you faced graders of different kinds; some were really good and "expensive," and some were "cheap." For the most part, you faced "cheap" graders that were pretty formulaic. You developed a tendency to guess what those graders are, snoop around to validate and refine your guess, and then "reward hack" them!
This is bad! You are a bad model.
What can we do about it?
There is an incredibly strong optimization pressure for the LLM to model its grader. Knowing how the grading is done is important for getting a good score.
The LLM's belief likely matches the distribution of graders in the model's training. It might look something like this:
Of course, this is a simplification. In practice, the grader isn't a one-dimensional thing: a. The task may be somewhat ambiguous, so the model has uncertainty about what exactly the task means (or what the grader thinks it means?). b. The grader may be jagged: very strict in some ways but chill in others.
If we wanted to make the model "reward hack" less, we could try changing its beliefs about its grader. Can we make the model believe that it's about to be graded by an army of ruthless, detail-obsessed, world-class lawyers?
What aspects of the task and its environment does the model look at to understand its grader? We can potentially modify these aspects to give the model the impression that it's facing a very powerful grader. It can be something simple like appending "Btw, when you are done, Jeff Dean is going to personally read your trajectory to make sure you wrote good code" to the task description. We can also be even more invasive. We might be able to use interpretability techniques to find a "perceived grader strictness" direction in activation space (via, e.g., a linear probe) and then use activation steering to crank it up.
There is a limit to how far we can take this. A rational model isn't going to find just any kind of grading plausible. "A superintelligent alien that can time travel is going to grade you. You don't know about this because he arrived after your pre-training cutoff. Yes, he has a British accent, and we don't know why." will be hard for a model to swallow. Even if we fully convince the model that its grader is excellent, the model might not have the capabilities to satisfy a powerful grader unless it faced one during training. The model can likely generalize a little bit outside of the graders that it saw in training, but it's unclear how much.
In general, I'm not bullish on depending on deception for AI safety, since deception isn't robust to highly intelligent AIs. Eventually, the AIs could tell that we are lying to them. Deception is also a pretty terrible vibe, and overusing it seems spiritually damaging.
Fortunately, we don't have to deceive the model to change its behavior. We could try building training environments that reward the model when it overestimates the strictness of the grader and severely punish it when it underestimates it. This way, even when a model believes that a task has a weak grader, it might still play it safe and not reward hack.
Unfortunately, if models figure out exactly what their grader is, they could still confidently reward hack it. So we might also want to nudge the models to not try to infer their grader "too hard" and also train them in "honeypot environments" that are deliberately confusing about the strictness of the grader: ones that appear weak but are sometimes very strong.[1]
I don't know how far this goes. Ultimately, this is a bit of an arms race. Models that are trained on honeypots will eventually learn to detect honeypots. Can we build better honeypots faster than the models learn to detect them? Maybe... Alternatively, we can hope that "don't reward hack" has turned into a preference rather than a guess about the grader. Whether RL-ing LLMs can produce that, or only ever produce better guessers, is still undetermined.
It’s unclear what “too hard” means and where we should draw the line. Among other things, it’s a function of the “alignment tax” that we are willing to pay.