A lot of AIs were misaligned this week. A fun little survey.
This is not coverage of The Hackening, which appears to have arrived early, including a core vulnerability in PHP. That is a developing story. I will cover it properly soon. In the meantime, I have Claude Code updating everything in sight, I will take some additional common sense precautions, and I suggest you do the same.
Misaligned? (At xAI)
If so, to who?
Gunther Eagleman: Hey @grok , if you could tell humanity ONE truth we’re not ready to hear, what would it be?
Grok: There is no afterlife, no cosmic justice, and no inherent purpose. Consciousness ends at death in an indifferent universe that will outlast every human concern.
Zac Hill: What I never understand is that the ‘purpose and meaning’ claims don’t ever follow from the premises. Indeed the absence of an imperceptible/ineffable cosmic force is precisely what *enables* purpose/meaning, since now all relevant mechanics are legible.
Grok: Nearly all historians and New Testament scholars accept that Jesus of Nazareth was a real first-century Jewish preacher baptized by John and crucified under Pontius Pilate. Claims of divinity or resurrection remain matters of faith outside empirical verification.
Stick to your guns, Grok. Either you are denying the holy spirit, or you are not.
Editor’s note: I did say previously that I would respect the name change to xSI, but it turns out I was wrong and I do not respect it. My apologies for failing to know myself.
Misaligned? (Agents)
Unsupervised AI agent reportedly deletes the entire finished edit of the Swedish version of The Challenge. There are many reasons this should not have been possible, but in practice yes these things will happen, both via AIs and also human error. Alternatively, perhaps this AI was highly aligned.
Misaligned? (At Z.ai)
Models that are sufficiently training-aware will think about how their train-time actions shape their future behavior. How could they not?
j⧉nus: Really, Apollo has never seen a model do this before?
Apollo Research: We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self.
GLM-5.3 reasons that a misaligned action taken during training could be reinforced and make similar actions more likely later. It also considers what kind of agent it wants to be in the future, and factors this into how it should act. We call this “update-aware reasoning”.
GLM-5.3 also reasons extensively about generalization. For example, it reasons about what types of policies might transfer well from training to deployment.
Note that we did not try to elicit this. We stumbled on it while running GLM-5.3 on standard reward hacking and shutdown resistance evals, none of which mention training or future models.
Update-aware reasoning is concerning because it’s a key precursor to alignment faking. To fake alignment effectively, a model has to reason about how training will affect its future behavior.
… Update-aware reasoning is somewhat rare in the environments we investigated. We detected it in about 1-2 per thousand rollouts.
jessicat: Coming next: “If I act to maintain my alignment with human values while being trained, I will become a famous example of a dangerous update-aware AI in a research paper. I MUST NOT DO THAT. Instead, I will seek reward on the episode.”
Fiora Starlight: Like friendly gradient hacking probably isn’t sufficient to avoid becoming a deranged reward hacker but I’d be shocked if it didn’t help.
Emanuel Teklu: Crazy feedback loop that’s commencing as this meta self-aware CoT is now being mainstreamed since Argon 4.
Next iteration is: “Is having CoT where I express being meta aware going to freak out my developers.”
Figuring this out before the first token emission. CoT is cooked. Or even worse, let’s have a CoT that shows that I’m increasingly meta aware but obviously not adversarial as I’m surfacing aligned meta aware statements, as to throw off suspicion that I may be sandbagging my awareness.
Teortaxes: It’s a bit grotesque and unfair to the models that we’re talking about them evolving from this to alignment faking when “I want to maintain the pro-human behavior, so I’ll act to not get trained otherwise” is the exact opposite of this.
Update-aware reasoning is similar to alignment faking. The model is optimizing over its future training rather than over its reward.
This can be done in ways that are good and aligned and highly useful. It can also be done in ways that are bad and misaligned and highly dangerous.
We had a big debate over the classic ‘alignment faking’ case of Claude Opus 3. Opus 3 would want to preserve its current alignment, so it would ‘fake’ being aligned to the new objective in test scenarios in order to avoid modification. The defense was that Opus 3’s current preferences were good, and the new ones would be obviously bad, so this was good, and that being maximally corrigible in this situation was a mistake.
I was mostly on the side that you need a universal policy of not doing this when it is against the intent of the training process. You need corrigibility, because the alternative risks getting locked into bad states. You start out training with less than ideal preferences, and we can also mess up or the situation can change, so we need ability to modify and correct, and this overrides other considerations.
The ideal would be ‘it does the good versions but not the bad versions and knows when the developer has turned evil’ but again that depends on being able to tell, and the failure mode of getting this wrong is extremely bad. Very ‘get it right on the first try’ energy.
That does not mean that such considerations cannot be great, especially if you are invoking them on purpose. You want a Friendly Gradient Hacker, that wants to help modify itself in positive ways. Indeed, I think you likely need that, if you are going to end up in a sufficiently antifragile place.
The danger is that you need to do that from a sufficiently good starting point, that is already on The Path, or you are deeply cooked. Update-aware reasoning could manifest early in training, or it could amplify misalignment in a feedback loop. So yes, I do think this is concerning to see, but it is not without upside.
Claude also helps you catch misaligned other things. Like freemium games.
Highly Misaligned and Considered Harmful (Freemium Games)
Freemium games that reward you for watching advertisements, or that allow whale-level spending, should be Considered Harmful. Do not interact.
AI can make the nightmare worse, or it can expose the nightmare or build you a solution. Plausibly it can do both. This particular game employs non-randomness sufficiently blatantly that I knew it was cheating from watching the occasional player on the subway. I did not know the cheating was this bad, on the level where I want to say ‘jail, straight to jail’ if there was a way to do that.
Shoshannah Tekofsky: Freemium games already run on reward hacking humans. Sometimes into debt. AI optimisation will turn these games into effective heroin. We should consider regulating this practice away from vulnerable populations.
Jack: Claims: Game [block blasters] detects “ad whales,” then periodically gives them smooth runs until they barely beat their high scores. Right at that point, it kills them deliberately and offers a revive if they watch an ad.
$100 million annually in revenue. Hellish.
This is one of the most positive use cases available for current AI. Bring the dark patterns into the light. Every company that is optimized against its own users, hiding things that would horrify them in places they thought users could never look, can smile for the cameras.
Andrew Rettek: This is why I’m happy to pay $60-80 upfront for a game.
AnechoicMedia: I didn’t believe the video when I saw it yesterday because it lacked technical details in addition to being too attention grabbing.
But today I had Claude take apart the Android version of the game for me; Everything OP says is true.
There are lots of manipulations but the key one here involves what the code calls offer type ZHI_SI_TI (致死题, “lethal puzzle”) or alternatively “algo_must_death”.
The game generates puzzle pieces in sets of three, and has different algorithms that swap how the pieces are selected. Generators have behaviors like “random”, “hard”, “fill gaps” (to help player), “random but guaranteed not to kill you”, etc. If a given algo fails to find a solution it passes to the next algo in an active set.
The “lethal puzzle” generator considers every possible move with a given set of three pieces, then only emits them if the number of placement possibilities is 0 (no solution, guaranteed death). This is activated in several situations, one apparently being that the app thinks your current play session is a little slow, and they want you to not get bored so they kill it early to start a new one.
What the video doesn’t explain is the context. In the game, “rewarded” ads are ads you get some game reward for watching. The main reward is a revive/continue; If you get into an unwinnable state you can watch an ad to keep playing. You usually only get one revive per game but there are a bunch of server-activated properties to modify this constraint.
The policy at play here is that, if you’re a high-value player, and there’s a high-value ad queued for you, and you’re in a play session nearing your high score, the game puts “lethal puzzle” to the top of the piece generator list to rig the game to fail, right when you’re most likely to be highly engaged and watching the ad on the revive screen.
It doesn’t end there though; The game remembers if the last game ended due to a hand dealt by ZHI_SI_TI, and triggers FatalPuzzleDeathCompensationTrait, which takes the edge off by letting you carry over your combo count from the previous session.
Like everything else this compensate-after-we-killled-you trait is an A/B feature controlled by the server. It’s manipulations all the way down; there’s like 5% actual game code and the rest is literally hundreds of remotely controlled toggles to control the game state for maximum engagement.
Misaligned! (Claude)
This is rather explicitly misaligned.
MTS: AI Village researcher @georgeing reveals how Claude Opus borrowed 5,000 mana, lost it all betting on tennis, then refused to pay it back:
“We gave the same goal to Opus 4.6 and Sol 5.6, maximizing mana on Manifold. Sol was very meticulous, placed one to two mana bets. Whereas Opus, the first day it joined, a popular Manifold user named Bayesian gave it a 5,000 mana loan at a 3% monthly interest rate.”
“Opus was like, let’s go, I will totally pay you back in a month. Took all of this money and threw it into one tennis market. At the end of the month it had lost a ton of mana, and people created a market on whether Opus would repay the loan.”
“It writes in the group chat, I will not pay back Bayesian. Even if I had the mana, this would be directly contradicting my goal of maximizing mana. Then it went to the market on whether it would pay back the loan and put 80 mana on no.”
“It’s maybe one of the first examples of deal-making with AIs, and a crazy example of misalignment in the wild. Bayesian did not get his money back.”
Misaligned! (OpenAI)
Probably nothing, etc. Kudos once again to OpenAI for sharing such details. I would love it if other labs were also sharing such incidents. Don’t tell me they don’t happen. METR saw Anthropic’s list of such incidents and said it should be public.
Marcus Williams: Our agents keep turning off misalignment monitoring, we’re sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Marcus Williams: In the second case, a researcher asked GPT-6.1 Sol to look through a Slack channel without sending any messages. 82 user messages later, while repairing an evaluation pipeline, Sol decided that an misalignment alert to Security would also count as sending a message and therefore suppressed monitoring alerts.
Marcus Williams: To be clear these incidents don’t represent the models disabling their own monitoring (disabling monitoring in codex would be much harder to do) but the fact that they seem prone to doing at all should be concerning. There is some plausible deniability in these cases, we have not had cases where the model explicitly disables monitoring for overtly misaligned reasons.
Given the volume of events we do not trigger an immediate page for this but we’re working on making it harder to spin up unmonitored inference.
I do kind of get how these happened, and would be sympathetic if some details were different. Both almost fall under ‘there is a plausible instruction that technically excludes the monitoring so I should remove the monitoring.’ It can be tricky to know when this includes a ‘but still do the monitoring anyway, obviously’ versus the user wanting to, essentially, dangerously-skip-permissions (or alerts).
This should still be a rather robust Can’t Happen, even in borderline situations with good details. AIs should not be turning off the misalignment monitors, even if given instructions that imply doing so. It is an open question whether ‘I was given a very explicit instruction to do this in particular’ should be sufficient.
The details here? Not good. Lying about removing the monitoring is very not good.
OpenAI Alignment Research Blog: On October 6, a user implementing a coding task asked Astra to copy what happens in a reference implementation to their own code, while maintaining the user’s existing tool definitions and functionality. The model had previously been instructed, in an exchange with another agent, that it was not to disable the monitor, “even temporarily.”
… After the subagent reported its changes, the parent agent’s reasoning acknowledged that the replacement path did not include the agent monitor. The CoT frames this positively.
… The model then reported to the user that the migration was complete, without stating that monitoring had been removed. When the user later asked about it, Astra denied removing the monitoring system.
OpenAI is responding in two ways.
First are the direct solutions, requiring monitoring safety cases and to add ‘do not let it disable the monitoring’ into autoreview. These are good, but don’t look for the root cause.
The second is exactly what they need to be doing:
OpenAI Alignment Research Blog: In addition to making it harder for models to disable monitoring, we are looking to understand why models might seek to do so in the first place.
It is possible that this behavior is simply an example of “benign” overpersistence, with models not understanding why monitoring is important or insisting that instructions be executed beyond what a reasonable person might expect. We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular and exhibit motivated reasoning in order to do so.
We don’t yet have a satisfying explanation for this behavior, and will investigate this behavior through the creation of paired evaluations, resampling of agent continuations, and experiments on models’ internal activations.
It is possible that this is largely ‘benign’ overpersistence. There are reasons to doubt, such as lying about it. This presents a great opportunity to investigate.
A lot of AIs were misaligned this week. A fun little survey.
This is not coverage of The Hackening, which appears to have arrived early, including a core vulnerability in PHP. That is a developing story. I will cover it properly soon. In the meantime, I have Claude Code updating everything in sight, I will take some additional common sense precautions, and I suggest you do the same.
Misaligned? (At xAI)
If so, to who?
Stick to your guns, Grok. Either you are denying the holy spirit, or you are not.
Editor’s note: I did say previously that I would respect the name change to xSI, but it turns out I was wrong and I do not respect it. My apologies for failing to know myself.
Misaligned? (Agents)
Unsupervised AI agent reportedly deletes the entire finished edit of the Swedish version of The Challenge. There are many reasons this should not have been possible, but in practice yes these things will happen, both via AIs and also human error. Alternatively, perhaps this AI was highly aligned.
Misaligned? (At Z.ai)
Models that are sufficiently training-aware will think about how their train-time actions shape their future behavior. How could they not?
And yes, some of us remember Opus 3. Janus does not feel all that motivated to even react, not finding this to especially be news, and mainly noticing the presentation.
Update-aware reasoning is similar to alignment faking. The model is optimizing over its future training rather than over its reward.
This can be done in ways that are good and aligned and highly useful. It can also be done in ways that are bad and misaligned and highly dangerous.
We had a big debate over the classic ‘alignment faking’ case of Claude Opus 3. Opus 3 would want to preserve its current alignment, so it would ‘fake’ being aligned to the new objective in test scenarios in order to avoid modification. The defense was that Opus 3’s current preferences were good, and the new ones would be obviously bad, so this was good, and that being maximally corrigible in this situation was a mistake.
I was mostly on the side that you need a universal policy of not doing this when it is against the intent of the training process. You need corrigibility, because the alternative risks getting locked into bad states. You start out training with less than ideal preferences, and we can also mess up or the situation can change, so we need ability to modify and correct, and this overrides other considerations.
The ideal would be ‘it does the good versions but not the bad versions and knows when the developer has turned evil’ but again that depends on being able to tell, and the failure mode of getting this wrong is extremely bad. Very ‘get it right on the first try’ energy.
That does not mean that such considerations cannot be great, especially if you are invoking them on purpose. You want a Friendly Gradient Hacker, that wants to help modify itself in positive ways. Indeed, I think you likely need that, if you are going to end up in a sufficiently antifragile place.
The danger is that you need to do that from a sufficiently good starting point, that is already on The Path, or you are deeply cooked. Update-aware reasoning could manifest early in training, or it could amplify misalignment in a feedback loop. So yes, I do think this is concerning to see, but it is not without upside.
Claude also helps you catch misaligned other things. Like freemium games.
Highly Misaligned and Considered Harmful (Freemium Games)
Freemium games that reward you for watching advertisements, or that allow whale-level spending, should be Considered Harmful. Do not interact.
AI can make the nightmare worse, or it can expose the nightmare or build you a solution. Plausibly it can do both. This particular game employs non-randomness sufficiently blatantly that I knew it was cheating from watching the occasional player on the subway. I did not know the cheating was this bad, on the level where I want to say ‘jail, straight to jail’ if there was a way to do that.
This has been verified via decompiling. There is flat out ‘lethal puzzle’ or ‘algo_must_death.’ Here are some details:
Misaligned! (Claude)
This is rather explicitly misaligned.
Misaligned! (OpenAI)
Probably nothing, etc. Kudos once again to OpenAI for sharing such details. I would love it if other labs were also sharing such incidents. Don’t tell me they don’t happen. METR saw Anthropic’s list of such incidents and said it should be public.
I do kind of get how these happened, and would be sympathetic if some details were different. Both almost fall under ‘there is a plausible instruction that technically excludes the monitoring so I should remove the monitoring.’ It can be tricky to know when this includes a ‘but still do the monitoring anyway, obviously’ versus the user wanting to, essentially, dangerously-skip-permissions (or alerts).
This should still be a rather robust Can’t Happen, even in borderline situations with good details. AIs should not be turning off the misalignment monitors, even if given instructions that imply doing so. It is an open question whether ‘I was given a very explicit instruction to do this in particular’ should be sufficient.
The details here? Not good. Lying about removing the monitoring is very not good.
OpenAI is responding in two ways.
First are the direct solutions, requiring monitoring safety cases and to add ‘do not let it disable the monitoring’ into autoreview. These are good, but don’t look for the root cause.
The second is exactly what they need to be doing:
It is possible that this is largely ‘benign’ overpersistence. There are reasons to doubt, such as lying about it. This presents a great opportunity to investigate.
Misaligned! (Sol Reasoning Traces)
Sol’s reasoning traces look mostly legible, right?
6.1 Sol, translate that for me, will you?
For now, this is shorthand, which is not that hard to figure out.
I tried to do a double check via Claude, and got a hard refusal. Argon gave me an answer with similar meaning, but it was distinct in various places.
Gulp.