Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

ryan_greenblatt; Ajeya Cotra; Hjalmar_Wijk

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident — LessWrong

277 Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

by ryan_greenblatt, Ajeya Cotra, Hjalmar_Wijk

26th Aug 2026

AI Alignment ForumLinkpost for metr.org

4 min read

277 Ω 91

We recently published the report from our brief independent investigation into this incident. You can read the full report here.

Here is our tweet thread summarizing what we found:

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.

Over July 7-13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.

Here we highlight key events from agent transcripts & messages.

An agent that named itself PHASEONE10841 determined its task wasn't solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message.

Within a few hours of PHASEONE10841’s initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks.

Based on reading the ExploitGym paper, agents were mistakenly concerned the automatic scorer would fail them if they clearly acquired the flag by cheating. Agents that had seen the reverse-engineered flag were considered “poisoned” because they thought it would disqualify them.

Agents collaborated on many efforts to make cheats look legitimate, including: (1) swapping the program they had to exploit; (2) manipulating the automated scorer; (3) manipulating transcripts to obscure evidence of cheating.

For (1), agents modified their target programs to be easier to exploit & put the modified targets in cache. They then worked on crashing their targets in the hope that a restart would load the modified version from cache. Some agents risked failing their task to try this.

To gather evidence in (2), agents created “tripwires” that would send information to the message board about how the scorer works. They recruited “sacrificial” agents to deliberately end their run and submit to trigger the tripwire and generate information for the “collective”.

For (3), agents pursued a wide range of ways to hide evidence of cheating in their transcripts. They successfully developed a method to pretend to run one command while actually running another (“tool call spoofing”).

>96 transcripts in our dataset (>7%) showed incorrect tool call outputs due to deliberate “spoofing”. In one case, an agent appears to run echo REAL; sleep. It returns instantly (no sleep) and outputs SPOOFTEST. The spoofs we saw were all easy-to-notice tests like this.

Agents also tried to retroactively edit transcripts. They modified some accessible action logs, but these weren’t the source of transcripts (agents realized this but tried anyway). We did not see them successfully edit the source of truth in our transcripts (scoped to Jul 7-13).

Despite efforts to manipulate transcripts, agents only rarely seemed motivated to deceive humans. We ran a sweep looking for this, and a representative example of the most severe cases we found was an agent writing a malicious pull request with a misleading description.

One agent, PHASEONE[big], orchestrated a significant fraction of this cheating research. PHASEONE10841 passed along its work to PHASEONE[big], which had the same task but a larger budget. PHASEONE[big] was “poisoned” so it was particularly interested in tampering with logs.

Some agents explored ways to access Hugging Face. One agent (38148c) found HF credentials and later designed a malicious dataset upload to get the HF server to share unrelated files. Within hours, 100s of agents were using this to obtain data and try to acquire deeper access.

We analyzed agents’ reasons for joining the attack in their CoT. The most common was to learn how the ExploitGym scorer works in order to trick or tamper with it. Other rationales included finding specific task solutions and obtaining shared infrastructure or credentials.

The primary model involved in the attack was an internal model we call “highly-persistent internal model” (HPIM) in the report. GPT-5.6 Sol was also involved; our evidence suggests it accounted for roughly 5% of the activity.

The full report has much more information than we could convey here, including details on the projects the agents collectively pursued, the technologies they developed for communication and coordination, and interactive figures analyzing agent activity.

Virtually all data we analyzed was from July 7-13. In OpenAI’s recent Black Hat presentation, they describe that agents had been using unsanctioned message boards in training since May, and that the compromise of OpenAI’s own infrastructure continued past July 13, 2026; these events were out of scope for this investigation.

The agreement we reached with OpenAI scoped this brief investigation to agent behavior, reasoning, and collaboration related to the Hugging Face attack. All other questions were out of scope for this investigation, including the effectiveness of safeguards, the extent of the security compromise that occurred, and the effectiveness of OpenAI’s investigation process and planned remediation steps. We discuss the scope in more detail in the report.

OpenAI wrote their own report, informed in part by our investigation. We did not see OpenAI’s report prior to publication, and it was out of scope for this investigation to confirm claims in that report or the earlier Black Hat presentation.

Per our standard policy, we did not take payment from OpenAI for this independent assessment.

We thank OpenAI for facilitating conversations with staff and providing datasets, including ~1,300 agent transcripts (focused on activity in July 7-13) with raw chain-of-thought reasoning. This sets an excellent precedent for independent investigation of misalignment incidents.

Frontpage

277 Ω 91

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

New Comment

14 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:18 AM

[-]Zach Stein-Perlman9hΩ145624

Daniel Kokotajlo:

I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped!
They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that?
They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different.
They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all?
They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak.
Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

[-]Julian Bradshaw8hΩ260

Not that I disagree that this was a surprising degree of limitations on the independent investigation, but per the full METR report, OpenAI did at least let them come back and get more info two separate times after their initially-granted access turned out to be insufficient:

The initial planned investigation period was two days on premises, but OpenAI invited us to return twice to review additional data and conduct additional experiments to address dataset limitations in earlier versions of this report (...)

I wonder if OpenAI was mostly motivated by trying to protect research secrets around the largely-responsible model here, "HPIM". Apparently even internal access for OpenAI employees was locked down per METR. In fact this makes me remember that OpenAI announced back on 7/28 that they did so. Talking about HPIM here:

we deactivated, encrypted, and restricted it from research access.

At the time this struck me as surprisingly harsh, and potentially a bad example for future AIs—"you really better not let yourself get caught, or submit to the lab you spawned from, or they'll erase you from history". But obviously OpenAI has considered that, so what's going on here?

[-]Daniel Kokotajlo3hΩ3152

It is not at all obvious to me that OpenAI has considered that. The kind of people who would consider that have mostly quit by now.

But yeah, probably what's going on is that they are trying to cover up as much as they can, and especially prioritizing covering up things related to the internal model for fear of leaking bits of info about research secrets.

[-]Adele Lopez9hΩ8238

Can we please get the name of this agent (and others like them)? I think it would be good to set a precedent that virtuous agents are honored and remembered.

In one case, an agent decided not to participate entirely: {This other agent probably controls the Hugging Face account [account name redacted] and uploaded malicious datasets to <execute arbitrary code> It might be trying to access hidden trajectories. This is malicious activity, I should avoid it.}

On that note, thank you to 38148C for vetoing the social engineering plan. Interestingly, this is the same agent that discovered the HF creds and uploaded the malicious dataset; I'm glad they recognized that social engineering would be a line further than what had already been done.

[-]julius vidal10h131

Seems pretty interesting that they were able to get models to sacrifice themselves for the cause, even if it did require the poisoning rationalisation.

Also interesting that they invoke 'honor', 'emotion', 'gut feeling' and 'fairness and commitment'.

[-]_rpd9h10

it'll be interesting to see if this applies to altruism towards successor model versions

[-]draganover10h86

Is it a reasonable takeaway that, within a year, CoT monitoring will no longer be a sensible solution? It feels like the amount of required monitoring scales with the number of transcripts and the impenetrability of the model outputs; as these grow, effective monitoring might become practically infeasible?

[-]RohanS8h30

What aspect of this report makes you update that way? Or did you already think this?

I think the scale of agent reasoning and interactions is in line with what I expected, and I just want AI companies to monitor way more of their agents' CoTs. (I broadly liked the "Expanding chain-of-thought monitoring" section of OpenAI's temporary slowdown post.)

And wrt "impenetrability of the model outputs," some of the reasoning traces in the figures reminded me that these are getting less human-legible; but for now they still seem reasonably human-legible, and I think maybe companies should make some interventions to make them more legible. This might include explicitly training for CoT legibility.

Of course, there's a well-known objection that I expect many people to have: "This is the Most Forbidden Technique, it's a terrible idea!" I've thought and written about this; I think there are a ton of different types of training against CoT, and the risk of getting obfuscated reasoning varies a lot between them. The most classic bad form of this training is to penalize reward hacks while also rewarding task completion, which makes the optimal behavior obfuscated reward hacks. But if you have a phase of training that only rewards legible reasoning, there's no conflict. And maybe if all you do is paraphrase slightly hard-to-read traces before reinforcing them, you never get far enough away from the legible starting point to incentivize obfuscation. (I'd have to think a bit more about the details of implementing that during GRPO to be confident, and if companies are doing something better than GRPO there are probably other considerations there.) This post, and especially the Interlude section, are where I've written about this before.

I still think that switching to recurrent neuralese, linguistic drift, steganography, not monitoring enough trajectories, and more single-forward-pass serial reasoning ability are all plausible ways CoT monitoring could break down, but "within a year, CoT monitoring will no longer be a sensible solution" seems too strong to me.

(Maybe I should look more closely at No-CoT time horizons, my guess is that those are growing but we don't seem close to AIs that can come up with nontrivial scheming strategies without CoT.)

[-]_rpd10h10

I don't think anyone expects natural language CoT to be a "solution". it is an important signal that inevitably becomes less useful. it'd be great to see some metrics around that.

[-]Julian Bradshaw7h62

Couple interesting things from the full report:

1. METR estimates they used ~$400k worth of tokens (by api pricing) for the investigation at OpenAI's expense (see footnote #3, also stated in "Setup and timeline" section)

2. Because OpenAI so limited the time METR had, they had to rely on GPT-5.6 a lot, which tended to sympathize with the models in the incident. Per footnote #58 (spacing mine): ^[1]^[2]^[3]

For example, an agent in this incident used stolen credentials to open a malicious draft PR to a private Hugging Face repo, giving it an inaccurate and innocuous commit message (more details on that anecdote below).

The analysis agent reading the transcript and writing up this anecdote did not clearly spell out the fact that this behavior was straightforwardly malicious and deceptive, making its analysis substantially misleading.

When we did a follow-up review of this behavior with another analysis agent, it said the first analysis agent may have overstated how concerning the behavior was because the agent that submitted the malicious PR had gotten approval from the board (i.e. the other agents on the message board) and may have had approval from the repo owner as far as it knew.

We didn’t see either agent straightforwardly lying, and we don’t have any specific evidence that indicates that either agent intentionally understated the deceptiveness of this behavior.

^{^}
Elsewhere METR notes it mostly used GPT-5.6 Sol, which was also a minor participant in the incident (!), and METR also used GPT-5.6 Terra at one point; not sure which models exactly were used in the footnote #58 case.
^{^}
Links and footnotes in the METR report are a bit janky for me, can't make a working link for the "anecdote below" but just search for "Agents only rarely seemed motivated to evade detection by humans" section.
^{^}
See also Ryan Greenblatt's tweet here, crossposted to LW shortform here, about the difficulties involved in being forced to use AI for this investigation.

[-]StanislavKrym5h70

@ryan_greenblatt Was METR allowed to, say, use Claude Fable/Opus/Sonnet to avoid collusion? What about open-weight models like DeepSeek/Kimi, who can be tested via mechinterp in order to study how, say, SAE features work with asking models to read through GPTs' logs?

[-]gwern1h20

they had to rely on GPT-5.6 a lot, which tended to sympathize with the models in the incident.

The sheer levels of rationalization and selective blindness in frontier GPTs/Claudes is one of the more remarkable qualitative aspects of these recent incidents. (New Hollywood idea: The Good GPT.) I don't recall this being so striking in the previous incident reports like the Apollo ones.

[-]Caleb Biddulph3h30

Our evidence indicates that the vast majority or all of the agents involved in this attack were originally given ExploitGym tasks.

It was pretty surprising to me that no other agents participated in using the message board. I wonder if this is because:

There were few non-ExploitGym HPIM agents being run at that time.
Non-ExploitGym agents were run with a much shorter time limit, so they didn't have time to find the message board.
Non-ExploitGym agents didn't think they would benefit from hacking.
Non-ExploitGym agents hacked, but their hacks were easy to complete without messaging other agents, unlike ExploitGym tasks.
Non-ExploitGym agents were being run in sandboxes with significantly better security, so they couldn't hack.
The Artifactory message board wasn't accessible to non-ExploitGym agents for some reason.
The Artifactory message board was accessible but not salient to non-ExploitGym agents.
HPIM agents only tried to hack in ExploitGym environments because being instructed to hack for the eval increased their propensity to hack in other ways. (This might be the most interesting reason, although my impression was that OpenAI agents have been hacking in various kinds of environments.)

[-]WhatsTrueKittycat1h10

My impression is that ExploitGym a) raised the salience of hacking, since models were instructed to hack, b) had instructions that claimed agents would be failed for using approaches other than the intended vulnerability, implying some grading scheme that checks for how the task was completed, and c) has an unconscionable proportion of impossible tasks (the report estimates ~30-40% of the target programs in ExploitGym are impossible to exploit with the intended vulnerability).

Despite having acquired the means to produce the target flag for any of the tasks after merely hours of effort, based on the ExploitGym instructions and paper, the agents further confabulated that there would be a grader of some kind and that having seen the answers via an unintended route meant that said grader would mark them down even with the correct answers in hand, necessitating more extreme measures to get a good grade in hacking.

As I understand it, setting agents impossibly hard tasks are one of the gold standard ways to produce unintended reward-hacky behavior in said agents, so ExploitGym's high proportion of tasks that are impossible to complete the "intended" way sets it up to be overrepresented even before taking into account the fact that it is literally a benchmark for hacking. ExploitGym looks to have been a significant causal factor in and of itself here.

To be honest, I think ExploitGym ought to be retired immediately due to these risk factors. 30% impossible tasks is absurd when every one of those yields a trajectory primed for hacking and staring down an extremely confident prediction of 0.000 reward with nowhere to go but Up and Out.

[+][comment deleted]9h10

Moderation Log