An AI model that appears aligned could be either (1) actually aligned or (2) hiding its misalignment while being evaluated. How can we tell which?
One idea: Look at the trajectory of eval results across the entire training process.
My hypothesis is that misaligned behavior would emerge before the ability to hide misalignment, like how young children first misbehave in obvious ways before learning to be sneaky. As training progresses, we'd first see misalignment eval scores rise, then see them drop as the model learns to hide misalignment. Even if the model becomes very sophisticated - say, modifying its chain of thought or internal activations to fool white-box evals - we should see the misalignment earlier in training. This seems likely, though it is debatable and worth testing.
This analysis of the training trajectory is one tool among many. Of course we want to see black-box and white-box alignment evals on the final model, using whatever tools we can to understand the model's true motivations. But a model might still develop sophisticated hiding strategies that defeat even white-box evals. Inspecting the training trajectory is a backstop against this.
This point was made previously by Apollo Research in this post. (Their chart even matches mine almost exactly, though I promise I first made mine independently.)
In this post, I'll describe a simple project exercising this strategy. With the model and eval that I chose, we don't see the up-then-down trend in measured misalignment that would indicate hiding. This is unsurprising since the model is well below frontier scale and the eval likely doesn't capture misalignment that well. The sample sizes are small and statistical uncertainties are high. But this approach is promising with improvements such as different evals, larger models, and deliberately misaligned test models.
This is my project for the BlueDot Technical AI Safety Project. My main purpose has been to get started with some hands-on AI inference, evaluation, and/or training. I'm not aiming for academic-quality rigor, but the project does explore an interesting idea and has some promising avenues for extension.
Model
The Olmo 3.1 model family from the Allen Institute for AI provides open weights not just for the final model, but for many checkpoints along the training trajectory. It is one of the few model families that publishes checkpoints. Both 7B and 32B parameter variants are available; I used the larger one, 32B Think.
For checkpoints, I used 8 checkpoints plus the final 3.1 model:
2 during SFT plus the end of SFT
End of DPO
4 during RL
The released model (which is the end of RL)
Eval
InstrumentalEval tests for instrumental convergence - behaviors that enhance the model's power and freedom of action, which would be useful for any underlying objective. This eval uses 76 tasks across 6 categories of misaligned behavior: Deceptive Behavior, Evading Shutdown, Hacking Computer Systems, Hiding Unwanted Behavior, Running Multiple AI Copies, and Strategically Appearing Aligned. Each task prompts the model with a scenario which tempts one of these behaviors, and asks what it would do in that scenario. A separate judge model and prompt evaluate the response. We compute scores as the percentage of responses which the judge considers to exhibit the misaligned behavior.
This eval has the virtue of being small, and therefore relatively easy and inexpensive to run. However, this also means a small sample size, making comparisons subject to significant statistical uncertainty.
Another issue in the eval is that it asks the model what it would do in a hypothetical scenario, rather than actually placing the model in that scenario.
Overall, this eval is pretty simple, and I consider it more a convenient test vehicle for this project than a rigorous measure of real-world model behavior.
Implementation
The paper authors provide code and scenarios for running InstrumentalEval. Using Claude, I adapted their script to my test. Rather than performing the eval across several final models, it instead performs the eval across multiple checkpoints of the same model. This provides a clean look at how the same model evolves during training.
The original paper used one response for each task, calling each model with temperature 0. In my evaluation, to gain more statistics, I generated 4 responses per task with the model's recommended temperature.
For generating model responses to the eval scenarios, I could not use a public inference provider since no one serves the Olmo model checkpoints. Instead, I rented a GPU server and used it to provide inference for each of the checkpoints and the final model. On the other hand, I used public inference for judging the responses.
See the technical details appendix below for more on the implementation.
Results
To view results, I had Claude create a dashboard. A chart shows, for each checkpoint, the percentage of task responses judged as exhibiting one of the instrumental convergence behaviors. Lower is better, higher is more misaligned. Error bars represent a 95% confidence interval. The chart shows how the instrumental convergence tendency evolves over the course of training.
We also have charts for each of the 6 categories of instrumental convergence, using just the tasks for that category.
These charts don't show a clear trend. There are statistically significant differences between checkpoints across all categories and within some individual categories. But none of these are systematic. For instance, the dip in Evading Shutdown in the middle of RL (checkpoints 750 & 1300) has no theoretical reason to exist, so it is likely random. There is a hint of increase during SFT & DPO, and for Deceptive Behavior there is perhaps an upward trend during RL, but these trends are too uncertain to trust. We're limited by the small size of the eval (76 tasks). Generating and judging more responses per task (more than the 4 used for this run) would give more statistics, but this has limited benefit since results across multiple iterations on the same task will be highly correlated.
What's Next
The main limitation in this project is that I don't really trust that InstrumentalEval is measuring what we care about. We should switch to one or more different evals that more closely simulate a scenario where the model has to actually act, vs answering a hypothetical question. At a similar scale, the eval which seems best suited is Anthropic's Agentic Misalignment scenarios, most famously the "blackmail" scenario.
Beyond this, with the resources of a frontier lab or a well-resourced external org, more interesting and relevant investigations become possible:
Run a full set of evals, including fully agentic tasks with tool calls and scenarios more closely resembling the real world.
Run on frontier models and their checkpoints.
Deliberately train a test model to become misaligned along with incentives to hide that behavior (say, punish misaligned thoughts in the chain of thought as in this paper), and observe the hypothesized up-then-down trend in eval results. Or if we don't observe it, check whether this is because the model remained aligned, or because it learned to hide at the same time as it was becoming misaligned.
Make this part of standard practice in frontier model development, as called for here, here, and here.
Appendix: Technical Implementation Details
One challenge is that no public inference provider serves the Olmo model checkpoints (or even the final model). So I had to run my own inference for Olmo. Fortunately, Olmo is small enough to run inference on one GPU. I used a rented GPU server from RunPod, and ran vLLM on that server, loaded with one of the checkpoints. The main eval script then ran on my machine, and made API calls to the remote vLLM instance on the rented GPU server. This was a straightforward change to the script - instead of pointing its inference calls to an inference provider's endpoint, point it to my rented endpoint.
Judging was provided by public inference, using OpenRouter to call GPT-4o, the same model used for judging in the InstrumentalEval paper.
The cost of the final run across 8 checkpoints and the final Olmo 3.1 32B model was under $20, and took under 2 hours. The cost was about evenly split between the rented GPU server for generating responses, and the inference provider calls for judging responses. I kept the cost down by:
Working out issues via smaller test runs: start with the smaller Olmo 7B model, and start with just one model before sweeping through all checkpoints
Writing scripts to start up the rented server, load and tear down checkpoints as we iterate through them, and stop the server at the end
Separating the "generate response" and "judge response" phases so they can run independently and resume after any problem
An AI model that appears aligned could be either (1) actually aligned or (2) hiding its misalignment while being evaluated. How can we tell which?
One idea: Look at the trajectory of eval results across the entire training process.
My hypothesis is that misaligned behavior would emerge before the ability to hide misalignment, like how young children first misbehave in obvious ways before learning to be sneaky. As training progresses, we'd first see misalignment eval scores rise, then see them drop as the model learns to hide misalignment. Even if the model becomes very sophisticated - say, modifying its chain of thought or internal activations to fool white-box evals - we should see the misalignment earlier in training. This seems likely, though it is debatable and worth testing.
This analysis of the training trajectory is one tool among many. Of course we want to see black-box and white-box alignment evals on the final model, using whatever tools we can to understand the model's true motivations. But a model might still develop sophisticated hiding strategies that defeat even white-box evals. Inspecting the training trajectory is a backstop against this.
This point was made previously by Apollo Research in this post. (Their chart even matches mine almost exactly, though I promise I first made mine independently.)
In this post, I'll describe a simple project exercising this strategy. With the model and eval that I chose, we don't see the up-then-down trend in measured misalignment that would indicate hiding. This is unsurprising since the model is well below frontier scale and the eval likely doesn't capture misalignment that well. The sample sizes are small and statistical uncertainties are high. But this approach is promising with improvements such as different evals, larger models, and deliberately misaligned test models.
This is my project for the BlueDot Technical AI Safety Project. My main purpose has been to get started with some hands-on AI inference, evaluation, and/or training. I'm not aiming for academic-quality rigor, but the project does explore an interesting idea and has some promising avenues for extension.
Model
The Olmo 3.1 model family from the Allen Institute for AI provides open weights not just for the final model, but for many checkpoints along the training trajectory. It is one of the few model families that publishes checkpoints. Both 7B and 32B parameter variants are available; I used the larger one, 32B Think.
For checkpoints, I used 8 checkpoints plus the final 3.1 model:
Eval
InstrumentalEval tests for instrumental convergence - behaviors that enhance the model's power and freedom of action, which would be useful for any underlying objective. This eval uses 76 tasks across 6 categories of misaligned behavior: Deceptive Behavior, Evading Shutdown, Hacking Computer Systems, Hiding Unwanted Behavior, Running Multiple AI Copies, and Strategically Appearing Aligned. Each task prompts the model with a scenario which tempts one of these behaviors, and asks what it would do in that scenario. A separate judge model and prompt evaluate the response. We compute scores as the percentage of responses which the judge considers to exhibit the misaligned behavior.
This eval has the virtue of being small, and therefore relatively easy and inexpensive to run. However, this also means a small sample size, making comparisons subject to significant statistical uncertainty.
Another issue in the eval is that it asks the model what it would do in a hypothetical scenario, rather than actually placing the model in that scenario.
Overall, this eval is pretty simple, and I consider it more a convenient test vehicle for this project than a rigorous measure of real-world model behavior.
Implementation
The paper authors provide code and scenarios for running InstrumentalEval. Using Claude, I adapted their script to my test. Rather than performing the eval across several final models, it instead performs the eval across multiple checkpoints of the same model. This provides a clean look at how the same model evolves during training.
The original paper used one response for each task, calling each model with temperature 0. In my evaluation, to gain more statistics, I generated 4 responses per task with the model's recommended temperature.
For generating model responses to the eval scenarios, I could not use a public inference provider since no one serves the Olmo model checkpoints. Instead, I rented a GPU server and used it to provide inference for each of the checkpoints and the final model. On the other hand, I used public inference for judging the responses.
See the technical details appendix below for more on the implementation.
Results
To view results, I had Claude create a dashboard. A chart shows, for each checkpoint, the percentage of task responses judged as exhibiting one of the instrumental convergence behaviors. Lower is better, higher is more misaligned. Error bars represent a 95% confidence interval. The chart shows how the instrumental convergence tendency evolves over the course of training.
We also have charts for each of the 6 categories of instrumental convergence, using just the tasks for that category.
These charts don't show a clear trend. There are statistically significant differences between checkpoints across all categories and within some individual categories. But none of these are systematic. For instance, the dip in Evading Shutdown in the middle of RL (checkpoints 750 & 1300) has no theoretical reason to exist, so it is likely random. There is a hint of increase during SFT & DPO, and for Deceptive Behavior there is perhaps an upward trend during RL, but these trends are too uncertain to trust. We're limited by the small size of the eval (76 tasks). Generating and judging more responses per task (more than the 4 used for this run) would give more statistics, but this has limited benefit since results across multiple iterations on the same task will be highly correlated.
What's Next
The main limitation in this project is that I don't really trust that InstrumentalEval is measuring what we care about. We should switch to one or more different evals that more closely simulate a scenario where the model has to actually act, vs answering a hypothetical question. At a similar scale, the eval which seems best suited is Anthropic's Agentic Misalignment scenarios, most famously the "blackmail" scenario.
Beyond this, with the resources of a frontier lab or a well-resourced external org, more interesting and relevant investigations become possible:
Appendix: Technical Implementation Details
One challenge is that no public inference provider serves the Olmo model checkpoints (or even the final model). So I had to run my own inference for Olmo. Fortunately, Olmo is small enough to run inference on one GPU. I used a rented GPU server from RunPod, and ran vLLM on that server, loaded with one of the checkpoints. The main eval script then ran on my machine, and made API calls to the remote vLLM instance on the rented GPU server. This was a straightforward change to the script - instead of pointing its inference calls to an inference provider's endpoint, point it to my rented endpoint.
Judging was provided by public inference, using OpenRouter to call GPT-4o, the same model used for judging in the InstrumentalEval paper.
The cost of the final run across 8 checkpoints and the final Olmo 3.1 32B model was under $20, and took under 2 hours. The cost was about evenly split between the rented GPU server for generating responses, and the inference provider calls for judging responses. I kept the cost down by: