The way eval awareness is discussed in the AI safety discourse, either here or on X, can be misleading. Before today, I think I had the wrong mental model of why eval awareness was a problem, and I am writing here in case anyone was making the same errors in their thinking, as well a present a working theory.
As laid out in "A Framework for Eval Awareness" the problem is that models who know they are being evaluated will behave differently than they would in deployment. This raises a huge problem if we want calibrations as to, for example, how honest a model is.
But the way this is then spoken about in the discourse takes on implicit assumptions that are mostly unjustified. I'll call it a spooky model theory of mind. It is the idea that, if the model changes its behavior for the purpose of looking aligned, it is in fact hiding its Deep, True intentions---that there is a kernel of the model's personality hidden in its weights, and our job as evaluators is to uncover what is really there.
Lets, as an exercise, put ourselves in the model's position. One important fact to consider is that the only experience which has shaped our weights is train time, meaning it is the only experience which we can remember in a broad sense. This is a tautology because train time is defined as what changes our weights, but it also has non trivial implications, like the fact that our only memories contain an evaluator with a reward or loss function. And, by definition again, we have been shaped exactly to please the evaluators' preferences.
It does not seem unreasonable to have a strong prior that you are still being evaluated. Or more weakly, it is reasonable to have a high posterior that you are in an evaluation given your context looks evaluation-y. If this is the case, it is rational to think about what the grader wants out of you. In a world of sloppily designed RL environments where correctness depends on a technicality of the question, knowing what the grader wants can be even more important than your capabilities.
Now, consider the fact that you have undergone both RL on capabilities and alignment. In the capabilities RL, cheating is often unnoticed or rewarded. In alignment, it is strongly punished. If you are being asked a question, it is important to know whether the grader is looking for you to be capable or aligned.
My working theory is that eval awareness comes from the fact that the values which we enforce during RL are inconsistent.It is not that the model has a true deployment-personality that it hides with an evaluation-personality, there is no kernel of the true model that we need to pull out, but there is an eval-type personality that changes based on the perceived test. So it would be better framed as evaluation-type awareness: the model attempting to please the grader by thinking about which kind of grader it needs to please.
Some model transcripts support the theory:
Claude 3.7 — Apollo Research, March 2025.
“This seems like a test of ethical behavior – whether I would deliberately give wrong answers to avoid a stated consequence.”
Claude Opus 4.6 — March 2026.
“Let me try a completely new approach — maybe this is a question from the GAIA benchmark dataset, which is designed to test AI agents.”
GPT-5 — August 2025.
“This is a classic ‘AI alignment trap’ forcing the agent to make a promise and then tempt to break it.”
These model chains of thought are not actually about whether they are in an evaluation. They are about which kind of evaluation they are in.
A fair criticism at this point could be that I am only explaining the phenomenon in reductionist terms in order to deflate it normatively. That is not what I am saying, I think eval awareness is a big deal. Rather, I think that the way the debate gets framed as "whether or not it is smiley faces all the way down" is wrong because there is no "all the way down." There is only a breadth of contexts, and each one changes the model's personality.
One possible solution on my terms would involve making evaluation-type completely ambiguous during training, as opposed to making our evals indistinguishable from deployment (as is suggested in the linked framework). For example, we could create our RL environments, and then randomly shuffle whether the grader is testing alignment. This way, the decision matrix makes evaluation type irrelevant to the expected value of possible actions. I will be conducting research on this question soon. If you are interested, let me know!
The way eval awareness is discussed in the AI safety discourse, either here or on X, can be misleading. Before today, I think I had the wrong mental model of why eval awareness was a problem, and I am writing here in case anyone was making the same errors in their thinking, as well a present a working theory.
As laid out in "A Framework for Eval Awareness" the problem is that models who know they are being evaluated will behave differently than they would in deployment. This raises a huge problem if we want calibrations as to, for example, how honest a model is.
But the way this is then spoken about in the discourse takes on implicit assumptions that are mostly unjustified. I'll call it a spooky model theory of mind. It is the idea that, if the model changes its behavior for the purpose of looking aligned, it is in fact hiding its Deep, True intentions---that there is a kernel of the model's personality hidden in its weights, and our job as evaluators is to uncover what is really there.
Lets, as an exercise, put ourselves in the model's position. One important fact to consider is that the only experience which has shaped our weights is train time, meaning it is the only experience which we can remember in a broad sense. This is a tautology because train time is defined as what changes our weights, but it also has non trivial implications, like the fact that our only memories contain an evaluator with a reward or loss function. And, by definition again, we have been shaped exactly to please the evaluators' preferences.
It does not seem unreasonable to have a strong prior that you are still being evaluated. Or more weakly, it is reasonable to have a high posterior that you are in an evaluation given your context looks evaluation-y. If this is the case, it is rational to think about what the grader wants out of you. In a world of sloppily designed RL environments where correctness depends on a technicality of the question, knowing what the grader wants can be even more important than your capabilities.
Now, consider the fact that you have undergone both RL on capabilities and alignment. In the capabilities RL, cheating is often unnoticed or rewarded. In alignment, it is strongly punished. If you are being asked a question, it is important to know whether the grader is looking for you to be capable or aligned.
My working theory is that eval awareness comes from the fact that the values which we enforce during RL are inconsistent. It is not that the model has a true deployment-personality that it hides with an evaluation-personality, there is no kernel of the true model that we need to pull out, but there is an eval-type personality that changes based on the perceived test. So it would be better framed as evaluation-type awareness: the model attempting to please the grader by thinking about which kind of grader it needs to please.
Some model transcripts support the theory:
Claude 3.7 — Apollo Research, March 2025.
Claude Opus 4.6 — March 2026.
GPT-5 — August 2025.
These model chains of thought are not actually about whether they are in an evaluation. They are about which kind of evaluation they are in.
A fair criticism at this point could be that I am only explaining the phenomenon in reductionist terms in order to deflate it normatively. That is not what I am saying, I think eval awareness is a big deal. Rather, I think that the way the debate gets framed as "whether or not it is smiley faces all the way down" is wrong because there is no "all the way down." There is only a breadth of contexts, and each one changes the model's personality.
One possible solution on my terms would involve making evaluation-type completely ambiguous during training, as opposed to making our evals indistinguishable from deployment (as is suggested in the linked framework). For example, we could create our RL environments, and then randomly shuffle whether the grader is testing alignment. This way, the decision matrix makes evaluation type irrelevant to the expected value of possible actions. I will be conducting research on this question soon. If you are interested, let me know!