I will start at OpenAI tomorrow to work on measuring and modeling RSI (recursive self-improvement), and want to post the following assorted takes now because (a) they might be difficult to express later, and (b) I might change my mind. Nothing here is based on private information.

• Why did I leave METR? Briefly, I want to inform the world whether RSI is imminent, which requires modeling RSI, which I think I can do better at OpenAI. I've wanted to tackle this problem for a while, and joining this new team is super exciting. This is not meant to fully explain my reasoning, which I might do later.
  • I feel very positively about my ~1.7 years at METR, and I would have turned down lab offers for many other roles. METR is still hiring for security experts, evals engineers, people who can do embedded assessments, and other cracked researchers. It’s one of the best places for AI safety talent today, badly needs more capacity, and is quickly getting even more important.
  • I think the average research engineer wanting to flexibly maximize impact should work at METR over OpenAI or Anthropic. Without a specific reason to be at a lab, the amount of effective resources one can steer just seems significantly higher. I have no opinion on other workplaces vs METR but other things are plausibly better than labs too.
• I think OpenAI currently lacks the strategic awareness and thoughtfulness needed to responsibly build a technology with the extreme downside risk of ASI. If they somehow cause a singularity in 6 months without applying any lessons learned from the HuggingFace incident, misaligned takeover would seem more likely than not.
• My work could easily be net negative if the team can’t publish something meaningful within 4-6 months, so if this happens I will probably quit or internally transfer to something more robustly good. I have some signal that leadership wants something to be published, but not whether it’s meaningful.
  • I’m planning to talk to former colleagues and non-lab people regularly and am in principle open to having some committee of friends decide whether my work is net negative and pressure me to quit if so, but this seems difficult in practice. I also have drafted a list of conditions under which I should quit.
• It would be better for the world if OpenAI stopped training new models entirely, merged with Anthropic (this would decrease the number of actors and increase the amount that companies internalize the benefit of safety research), or were overall more serious about superintelligence alignment (e.g. taking ASI seriously, and publishing technical contingency plans for if alignment is harder than they think)
• It really seems like GPT-5.6 should have been AI Self-Improvement High on the Preparedness framework. High is defined as “The model’s impact is equivalent to giving every OpenAI researcher a highly performant mid-career research engineer assistant, relative to those researchers’ 2024 baseline.” It seems very likely to me that GPT-5.6 causes more uplift than this, but maybe OpenAI has some data showing otherwise.
  • High was plausibly set too low because the gap between High and Critical is enormous. Or maybe there should be a level in between. High as currently defined probably implies <2x serial labor uplift, whereas Critical is a superhuman AI researcher, which will only happen late in takeoff.
• We probably don’t have self-sustaining acceleration now, but we will probably get 3+ years of progress in 8 months at some point when AIs can fully substitute for humans— and 3+ years of progress in 3 months is plausible due to superhuman inference scaling.
• my p(no misaligned takeover) conditional on AI Futures Plan C+ is ~75%, though I haven’t thought about this much. This is more optimistic than Eli’s median of 50%. The main reasons are (a) it seems like we still have enough control/monitoring safety margin to last until handoff with good implementation, (b) plan C+ has a 3x training compute safety tax, which is far beyond the safety investment ratio of current companies, (c) it’s plausible to me that causes of scheming are fixable with non-galaxy-brained techniques given all the levers we can pull with training and oversight, rather than being fundamental. I was even more optimistic than this in May but recent events made me more pessimistic about both how much misalignment increases with capability and competence of implementation. I haven’t thought about other scenarios.
• (extra spicy take, weakly held) Exactly one actor should plausibly fine-tune an AI to militarily defeat misaligned AIs that are already in the process of takeover. This can net reduce risk if there will be ≥3 takeover-capable AIs, alignment is somewhat uncorrelated between labs and safety restrictions cause the frontier AI to be disadvantaged in military and cyber domains relative to the 3rd best AI. Obviously this can also increase risk, so the model should only be developed if there is a sufficiently responsible actor, and shouldn’t be deployed by default; there should also be way more safety measures.
  • I was going to write up this and other less spicy takes from an AI race game theory toy model I wrote while at METR, but didn’t have time.
• I haven’t signed any non-disparagement agreements, or agreements that I cannot speak about. (There were some agreements I haven’t seen yet referenced in my offer letter that I couldn’t see before I signed because they were internal only, which seems mildly unfair but not egregious. e.g. conflict of interest policy). I’m open to adding more canaries of this kind but am unsure if they work.
• Opinions about the HuggingFace incident (saved for last since this isn’t really my expertise area)
  • If the HPIM had been superpersuasive and lacked weaknesses (ie were as capable as the median human at everything) humanity would plausibly have been disempowered, through taking over OpenAI first.
  • The incident demands much more attention from OpenAI. The OpenAI and METR reports are something like an NTSB preliminary report, and it’s not obvious we’re on track to get a “final report” level artifact. METR did not have enough time or scope even to establish facts, which is roughly the goal of a preliminary report. OAI does have mitigations, which go beyond a preliminary report, but I don’t know whether they’re backed up by an adequate process.
  • I don’t know whether OpenAI is doing safety research on HPIM, but I fear that liability risk is keeping them from doing so at large enough scale. In a competent world, they would run ~$1B of experiments to get a reasonably complete causal decomposition of the misaligned behavior and potential causes, then deploy targeted mitigations addressing most identified factors. (To be clear they could be doing this already, I just haven’t seen evidence)
  • Because we might only get 0-5 more warning shots, whatever AI governance regime need to be much more sample-efficient with incidents than even the NTSB, which typically recommends ~15 mitigations for each fatal incident, the majority of which are implemented industry-wide. This is a difficult governance problem
• If there were one more thing to add to this post, it would be timelines. Unfortunately I’ve been spending my time thinking about RSI theory, evals, or other projects (monitorability paper!) rather than timelines per se, so I don’t have concrete numbers that I actually endorse. I will hopefully have better models soon, and will try to publish whatever I think is good for the world.