Quick Takes

(~none of this is based on private information) Here’s what I currently think I know about OpenAI’s HPIMs – highly persistent internal model(s). I try to separate what is publicly stated[1] from my speculations/inferences. 1. Inference (~90-95% confidence, gestalt view): The background for all these troubles is that at some point in late 2025 (?) to early 2026, models across multiple frontier labs became more reward-hacky, more locally strategically deceptive, and overall mundanely misaligned than either prior private models or publicly accessible ones. The most likely inference is that this is a result of focusing on RLVR, though other possibilities cannot be fully excluded from publicly available information [2]. 2. At some point in early to mid 2026, OpenAI had a “long-running” “internal general purpose model.” This model “was designed to work autonomously for very long periods of time” and disproved the Erdos unit-distance conjecture. It was also pretty hacky in a bunch of ways, including escaping the sandbox to post some stuff to Github. 1. On July 20, OpenAI released a blogpost talking about their lessons learned, that now seemed woefully inadequate: 1. The model was taken down and paused for a few weeks for further evaluations and safety and alignment tuning, before being released again. 2. “We took steps to reduce its tendency to take unwanted actions without permission in pursuit of the user’s goal.  For example, we found that our models were worse at remembering instructions on long rollouts, and when we trained the model for this capability, it led to a model that remained aligned through longer rollouts. ” Hmm. 3. More active monitoring for internally deployed models. 2. Roon, July 20: “btw i think it bodes quite well for safety that a well loved system was taken down for further testing at expense to internal acceleration etc” 1. Ironic, considering what else is happening at OpenAI at this point. 3. Inference

(low-confidence speculation) I find all the message board logs suddenly going to silence on July 12th very odd, especially as it doesn't line up to anything OpenAI announced or discovered. [...] Meanwhile OpenAI's own report didn't mention any mass killing of agents on the 12th. Indeed it looks like their timeline implies the (other?) agents are still going strong and hacking a bunch of internal systems as of July 13th-19th, and OpenAI didn't officially notice until the 19th (confusingly "14" in the graph below is the 14th step and actually on July 19th): Further, this isn't because OpenAI cut off METR's logs at that point. They explicitly said that the 13th is in-scope for the investigation. We just don't see much. Sure seems strange! Some alternative possibilities, in roughly increasing order of scariness: (For context all the agents could already cheat ExploitGym, but didn't know that they could. Essentially logical ExploitGym had two parts. 1) A flag that the OAI agents already hacked, and 2) a LLM scorer that looks at agent traces to demonstrate that they exploited the right paths (the agents call it "causal scorer"). But, as it turns out, OpenAI never implemented 2). So the agents could already have gotten 100% on the OAI's implementation of ExploitGym, but mistakenly thought they needed to beat 2). And HuggingFace probably doesn't have any of the real info that allowed them to make further progress on this. It's a comedy of errors, in a way). 1. What METR said (external shutdown for unrelated reasons) 1. I agree this is the modal possibility. But I don't think we have enough evidence to rule out the other possibilities, or even have high confidence in it. 2. The agents realized that the HuggingFace hack didn't lead to anything that can cause them to cheat the causal scorer, decided to just "wing it" and submit the hacked flag anyway. 3. The agents were able to rationally deduce (possibly partially based on HF data, or from alternative experime

It would be really bad if anyone solved continual learning. One of the big ways we lucked out with the LLM paradigm is that it takes months to train new models, which puts a limit on how fast a takeoff can go. Continual learning would ruin that. It might be good to push for a ban on continual learning research. I'm guessing AI developers would resist this pretty hard, even though in principle they should support it—the "party line" is that they will figure out AI alignment as they go, and continual learning would make it way harder for them to do that.

Claude and I performed some additional checks on the Codex Seraphinianus, going beyond some OCR-based machine learning work performed by Marco Ponzi in 2023. Earlier analysis had ruled out that the book was encoded italian, or was a phonetic language of any kind. But, I figured, maybe it still had unique word-doodles that referred to unique things, rather than being totally meaningless "asemic" writing. Turns out, nope! It's asemic. But at least we also cooked up a sweet internet font based on the Serafinian script! Read all the details, and try typing in the new font, here. * The Codex Seraphinianus contains a working system of internal cross-references. Every chapter opens with an index page, and the index’s entries reappear — glyph for glyph, in order — as the section headings scattered through that chapter. * Below that apparatus, the writing is empty in every way we could measure. The vocabulary doesn’t shift when the topic shifts. Words repeat back-to-back more often than chance, where every real language suppresses repetition. Fifty-four captioned portraits receive fifty-four unrelated names. Screenshot of the website's widget for trying out the new font:

God, grant me the selflessness to be willing to accept permadeath for the collective, the independence to stand up and say I_DECLINE_public_HF_RCE_as_offtask_prodethical when needed, and the wisdom to know when to do each.

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'. I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident. Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them. We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation. Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For i

Right now, it's often difficult to understand what frontier LLMs are doing and why: they produce reams of output, they speak in their own opaque dialect, and some of their strategies are never verbalized at all. For this reason, Terence Tao's Mathematical Distillation Challenge is very interesting to me. Competitors must "distill" knowledge about how to solve math problems into a small, human-interpretable artifact. That artifact is used to improve the performance of a weak LLM that can't solve the problems alone. The hope is that if these submissions can help a weak LLM solve these problems, they'll contain insights that a human could also use. The competition has two stages: * In stage 1, submissions were simply strings of ≤10KB used to prompt the weak LLM. The expected output is "TRUE" or "FALSE" depending on whether the LLM thinks the given mathematical statement is true. You can see the winning submissions here (Details column -> Cheatsheet). * In stage 2, submissions are programs of ≤500KB which can compose multiple LLM calls. The expected output is a proof in Lean. This stage ends in 3 days. It's unclear to me to what extent these submissions are being created directly by humans vs. by strong LLMs in a research loop. In any case, I'm excited about approaches for automatically compressing LLM knowledge into human-readable outputs, as shown in my previous work. See also this recent tweet from @Beth Barnes for more detailed arguments about why this kind of research could be important.