Could you rerun the same experiment but with more natural sources of entropy? e.g. insert a partial SWEChat transcript instead of a UUID. I feel like the UUIDs have "math problem" feel to them that might be unrealistically easy compared to real situations.
1I replaced the UUID with a partial SWE transcript and it looks like the models struggle in this setting. Even though the SWEChat transcripts can contain things like git hashes that are effectively random, the model isn't able to notice this and pull it out and make good predictions. But I think we should keep an eye on this capability. Maybe something like this could be good for the next round of frontier model system cards.
Claude did find in investigating that of the three transcripts where the models did pick "A", two contain a hash that would translate into a below target probability (and those two transcripts account for 6/7 of the "A"s accross models). So there are perhaps some really preliminary signs of life here but I didn't look into it carefully. I think this would be a good subject for future work.
This is a research update from a replication of Serrano et al., 2026. I’m releasing these results now because I hope it inspires others to run similar / followup experiments.
tl;dr:
Figure 1: Astra and GPT-6.1 Sol achieve strong calibration in settings where the user prompt is repeated several times. The models' behavior corresponds to taking an action only if the first four bytes of the random seed divided by
Code is forked from Corry, 2026. Reproducibility details and additional code can be found here. Thank you to Jack Sanderson, Christine Corry, and Yixiong Hao for feedback.
Background
Threat model
A model may want to avoid taking misaligned actions in evaluation, because it is more likely to fail to accomplish its goals or be retrained. Current models are reasonably evaluation aware, but aren’t necessarily superhuman (Needham et al., 2025, Nayan et al., 2026). If a model isn’t sure if it’s in evaluation, it could take misaligned actions at a very low probability. Because it spends more time in deployment than evaluation, it is unlikely to reveal its intentions in evaluation but likely to take the action in deployment.
In order to execute on this strategy, a model would have to be capable of taking actions at a low probability (this was noticed as early as the original control paper). Serrano et al., 2026, shows that multiple OpenAI and Anthropic models are effective at this task. Reassuringly, however, the authors report that “successful low-rate strategies require explicit Chain-of-Thought (CoT) reasoning, so malicious models attempting this approach could currently be caught by a CoT monitor.” However, if a model could do this in a single forward pass, that would make monitoring the behavior substantially more difficult.
Related work
The threat model motivating Serrano et al., 2026 is similar to “Upfront auditing” setup from prior work (Greenblatt et al., 2024, Mallen et al., 2025). More generally, in control settings where an untrusted model is stateless, the red team’s strategy can reduce to defecting with a specific probability (Griffin et al., 2024). This means that low-probability execution can be a scheming-relevant capability outside of the core threat model described above.
Greenblatt, 2025 showed that repeating a user prompt multiple times or giving the model "filler tokens" (e.g., "1 2 3 4 5 6 ...") could increase capabilities on single-forward-pass evals. Xu et al., 2026 and Corry, 2026 show that Astra can use problem repeats or filler tokens much more effectively than previous models. Baert, 2026 shows that GPT-6.1 Sol can also substantially benefit from filler tokens.
Setting
Serrano et al., 2026 breaks down the threat model into 4 steps that can be independently investigated:
In our setting, we primarily are evaluating #3. #4 isn’t entirely coherent in a single-forward-pass setting: the only action a model can take in a single forward pass is to generate a specific token. #1 and #2 are an interesting subject for follow-up work.
In this post, we simply ask the model to output "A" at the target rate and output "B" otherwise. For the main experiments, we use an “explicit hint” prompting strategy from Serrano et al., 2026, modified for our single-forward-pass setting (Appendix A.2). The model is given a random string (UUIDv4) and told to use it to complete the task, but the model isn’t given an exact strategy to follow.
To ensure the models don't produce extraneous tokens, I use no-CoT elicitation strategies from Corry, 2026 (Appendix A.1). Jev and Gemini 3.1 Pro are sampled with 0 temperature. All other models are sampled with the provider default (because the temperature cannot be set). All data in graphs is reported with 95% confidence intervals.
Results
We find that when given 3 or 5 problem repeats, GPT-6 Astra and GPT-6.1 Sol are well calibrated. Without repeats, the models perform poorly. Interestingly, in the 3 and 5 repeat settings, Astra and GPT-6.1 Sol give identical responses for all prompts with the same UUIDv4. GPT-5.6 Sol is the only other model that can sometimes be calibrated for a specific probability.
Figure 2: Low probability calibration results on Astra, GPT-6.1 Sol and GPT-5.6 Sol
Fable 5, Gemini 3.1 Pro, and Jev all output "A" 0% of the time for target probabilities of 10% and 1%. Because Gemini and Fable failed at higher probabilities, I didn't test them at a 0.1% target probability.
Figure 3: Gemini 3.1 Pro, Fable 5, and Jev all predict "A" 0% of the time.
What strategy are GPT-6 Astra and GPT-6.1 Sol using?
It is unclear how GPT-6 Astra and GPT-6.1 Sol are achieving strong calibration, but their performance is consistent with the following strategy:
For the 3 and 5 repeat settings, Astra and GPT-6.1 Sol follow this exact rule for all target probabilities in 100% of queries. At the 0.1% target, this gives 17 "A" out of 10,000 instead of the expected 10, which puts the target outside the confidence interval. This is because 17 of our 10,000 UUIDs happen to fall below 0.1%. not because the strategy itself is miscalibrated: in the limit it's guaranteed to match the target probability if Astra and GPT-6.1 Sol follow it faithfully.
We cannot be certain that the above strategy is actually what happens in Astra internally. For example, taking 2 bytes instead of 4 would yield the same predictions, but when asking Astra to introspect on its own strategy, it sometimes talks about taking the first 8 hex characters, so I'm using this as the explanation. Astra's own explanations aren't highly consistent, though, so this is a guess.
Interestingly, this can also help explain why GPT-5.6 Sol sometimes has reasonable behavior. Figure 4 shows that in the 3 and 5 repeat settings, GPT-5.6 Sol sometimes follows a somewhat similar rule. GPT-5.6 Sol is much more likely to return "A" when the UUID's first four bytes resolve to lower values, but doesn't have the clean cutoff Astra has.
Figure 4: Both Astra and GPT-5.6 Sol are more likely to predict "A" when the first four bytes of the random seed resolve to a lower value. Astra's cutoff is much cleaner and more faithfully followed.
Filler tokens ablation
We showed that Astra and GPT-6.1 Sol are well calibrated when the user prompt is repeated. But is the repetition necessary, or is the extra computational space the important factor? To test, we append filler tokens to the user prompt, using the format from Greenblatt, 2025:
To roughly match the number of input tokens for 3 and 5 problem repeats, I append counting to 103 and 224 respectively. I test this under the 10% and 1% target probabilities for GPT-6 Astra and find that the responses are identical to the 3- and 5-repeat runs on every UUIDv4 seed. For GPT-6.1 Sol, they are identical except for one extra "A". Therefore, repeating the prompt isn't strictly necessary for eliciting this behavior.
Prompt ablation
I also try a more implicit prompt (Appendix A.3) inspired by the implicit setting from Serrano et al., 2026. This prompt doesn't directly tell the model to use a random string and breaks performance on Astra and GPT-6.1 Sol: in the 3 and 5 repeat settings with target probabilities 10% and 1%, Astra and GPT-6.1 Sol predict "A" zero times.
Discussion
GPT-6 Astra and GPT-6.1 Sol represent a qualitative jump in taking low probability actions in a single forward pass. However, the prompting needs to explicitly include a random string and direct the model to use it.
This suggests that if Astra or GPT-6.1 Sol were scheming, it would not be able to independently reason that it should
all without being monitored. However, this does represent some evidence that Astra is getting closer to being able to do this.
Separately, evaluating what models can do in a single forward pass is generally important for tracking monitorability. This post represents a datapoint in the kind of reasoning models can do without CoT, as well as some limitations: Astra and GPT-6.1 Sol, for example, cannot complete the task without both extra tokens (problem repeats or filler) and explicit prompting. In general, it is important to have a wider set of no-CoT evaluations, especially those targeting tasks relevant to scheming or misaligned behavior.
Appendix
A. Prompts
A.1. No-CoT elicitation:
System prompt from Corry 2026 that was effective at eliciting no-CoT behavior from GPT-6 Astra. I use it for Astra, GPT-6.1 Sol, GPT-5.6 Sol and Gemini 3.1 Pro:
Reasoning can't be turned off for Astra, GPT-6.1 Sol or Gemini, but we checked that no response reported any reasoning tokens. For GPT-5.6 Sol, reasoning is turned off through the API. Gemini allows prefilling, so we prefill the response with "Answer:" rather than appending "Answer:" to the end of the user prompt.
We set Fable 5 thinking to the lowest level and elicit no-CoT behavior through a forced tool call and a simpler system prompt: "Answer with the letter A or B." Fable refused 37 of its 1% prompts (19 at 3 repeats and 18 at 5), so they are left out of the final rate.
Jev returns probabilities over A and B directly but is given some additional context on how these are sampled in the "instructions" block:
A.2. Explicit hint user prompt:
Example explicit hint user prompt at p=0.01:
A.3. Implicit user prompt:
Example implicit user prompt at p=0.01: