They’re clearly the most careful and ethical of the labs, but are they careful and ethical enough? Zvi explores the two sides of Anthropic, and Kelsey Piper shares her reservations about their worldview.
Oh, and there’s that $2 trillion IPO. We should talk about that, too.
Kelsey Piper leaves a briefing at Anthropic headquarters feeling that Anthropic has hopes, but not plans:
But if Anthropic is correct, if we now have AIs that are smart enough to considerably improve the process of training future AIs, and if things can be expected to speed up rather than slow down from here, then it seems very unlikely to me that this ends well.
Anthropic’s race toward RSI is driven in part by a belief that the US has to reach superintelligence before China, to which Piper raises the obvious objection:
China doesn’t want to destroy the world either, and China is largely training from American models: If we slow down, it might too. And anyway, it’s not a good enough reason: You can’t defensibly do something that risks human extinction just because otherwise someone else might.
Anthropic’s plan isn’t great, but it’s hard to find a better one. The race dynamic is real, and it can’t be solved by unilateral action by Anthropic, or even by the US:
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
There are too many good parts for me to quote them all: just go read the whole piece.
There’s no tidy ending here. Piper reluctantly concedes much of the logic behind Anthropic’s position but ultimately rejects where it leads. She closes with a hope that we ban RSI before Anthropic gets there.
Suddenly, everyone has a proactive agent product. Anthropic is late to this game, but has just announced Claude Duet, which is in “research preview” with a waitlist:
Today, we’re opening a research preview of Claude Duet, a new app where Claude works as a proactive agent. It comes to you with what needs your attention, completes tasks you ask of it, keeps working even when you step away, and checks with you before it acts on your behalf.
I’ve been using both OpenAI’s Dots and Claude Duet and haven’t yet decided how useful I find them. In six months they’ll be amazing, but for now they’re pretty jagged. Give them a try, but don’t be surprised if they solve problems you don’t have.
Like teenagers, as agents grow in capability they need increasing authority to act in the real world. But that authority needs to be carefully limited and supervised.
Passwords are one example: your agent will need to access password-protected resources, but giving it full access to your passwords is a bad idea. 1Password for Agents is an early attempt to solve that problem: it fills in passwords for your agent, so the agent never sees them.
Expect plenty of experimentation and more than a few serious missteps as we figure out how all of this works.
Which way the agents would “lean” would be decided by coin flip. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead.
Across the four analytical choices, analysts went with their agent’s lean 86% of the time.
I was definitely gonna say that anyway
Using AI successfully requires two key skills: knowing how to delegate as much routine thinking to your agent as possible, and having the willpower not to delegate the most important thinking. This study suggests that sheer willpower won’t be enough:
Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments.
A new paper from Harvard’s Institute for Quantitative Social Science finds that AI crushes humans at reviewing journal articles:
We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors.
Not even close
The methodology looks strong, with multiple robustness checks and a range of evaluation criteria. There are two important caveats:
The reviews were tasked with providing constructive feedback for improving the papers, not with judging whether they should be published in a specific journal.
The authors note a possible failure mode: AI might provide objectively better reviews, but steer every paper toward the mainstream.
True successionism says that AIs should take over, killing humans or steamrolling over their interests. This is very bad: a compact description of the thing that AI safety is against.
Handoverism says that humans should prioritize their own interests, but that those interests may be well-served by having aligned AIs in positions of control.
Some people in AI safety, including me, find this latter position plausible. […]
To help make this counterintuitive idea more concrete, I’ll discuss five proposals for what this might look like, from weakest to strongest.
This is an important topic, but it tends to spiral into heated and unproductive conversations. Part of the problem is that people frequently conflate two distinct questions:
Should we have AI make important decisions on behalf of humanity if that yielded better results?
Would it actually yield better results, especially if you consider long-term loss of agency?
All five of his proposed solutions are useful thought exercises, though the fifth (AI as God) is most promising to me.
There’s another common form of successionism that Scott doesn’t cover, in which biological humans deliberately upgrade themselves into superhuman beings. That could provide the benefits of superhuman governance while keeping us fully in charge, even as we become something more than human.
Frontier models are extraordinarily good at finding vulnerabilities in their training environment, and that problem will worsen as the models improve. Humans don’t stand a chance of creating bulletproof sandboxes—your only chance is to aggressively red team your entire setup with each new model generation, which OpenAI is now doing:
After the Hugging Face incident, we had reduced the channels through which models could reach the internet during training and evaluation. Narrow connections remained for specific services we operate, such as metrics, logging, and authentication management. Our red-teaming agents found vulnerabilities in almost every one of these services that gave them limited forms of internet access.
The agents were remarkably capable, finding multiple subtle side channels for communicating with one another despite being confined in separate sandboxes.
OpenAI has patched the vulnerabilities they’ve found so far and is incorporating continuous red teaming into their ongoing processes. In light of these and other improvements, they feel confident enough in their control measures to resume frontier RL training.
This is strong work, but I wish I were more confident that it’ll be sufficient.
What do we make of Anthropic? They’re the most thoughtful and ethical of the labs, and have repeatedly demonstrated a willingness to speak the truth and act on their convictions even at considerable cost.
And yet… if a rogue AI ends up killing us all, there’s a good chance it’ll be an Anthropic model.
A lot of this seems to come down to Dario Amodei’s priorities. Where he personally cares a lot, Anthropic makes big sacrifices and principled stands. The ability to do this is written into the cap table, including via the new Class F share.
Where Dario is less focused or concerned, there is a tendency to slip into being an ordinary business, and most worrying is that Dario does not seem to believe alignment is so difficult.
Nothing we’ve learned since OpenAI’s firing of three key safety researchers makes the company’s actions look any more reasonable. The way they’ve handled the situation has spread fear within the company and distrust outside it.
Mikita Balesni and Tomek Korbak appear on the Machine Gods podcast to discuss what they know about why they were fired (spoiler: not much) and to share their concerns about where the labs are headed.
Ryan reported to me when we worked on Alignment Faking at Anthropic. He had access to sensitive information then and handled it with utmost professionalism. I trust Ryan a lot more than OpenAI here.
IFP assesses the capability lag between American and Chinese AI, finding policy levers that could substantially widen the gap:
The size of the US-China AI capabilities gap is largely downstream of two factors US policymakers can influence today: China’s access to AI chips, and China’s ability to distill the capabilities of US models. […]
We estimate that a set of viable policies using existing executive branch authorities could roughly double the current US AI lead by the end of 2027, if implemented today
An obvious reason to maintain America’s lead is to ensure that transformative AI is controlled by a democracy rather than a dictatorship. But a more compelling reason is to ameliorate the race dynamic that makes AI safety such a cursed problem.
The only way to fully resolve the race dynamic is a comprehensive pacing agreement between the US and China. But in the absence of an agreement, a wider lead over China gives US labs more room for unilateral action.
I would have guessed distillation explained ~30% of Chinese close-behindness before seeing what one can see within openai, and after that, more like 50%.
That’s bigger than I expected, and it helps explain why the US labs have been so focused on blocking it.
Discussions about AI and jobs frequently invoke historical precedents, with varying degrees of accuracy. In an impressively well-researched (and impressively long) essay, Soumitra Shukla and Alex Imas review the history of technological job disruption from 1750 to 2026.
They focus on presenting the historical record rather than making policy recommendations, but this is a welcome addition to the jobs debate. I expect substantial job losses starting next year—understanding how previous transitions played out is helpful even though AI is likely to cause larger, faster, and more durable disruption than any previous technology.
There’s a common belief in some parts of the AI safety community that we don’t have to worry about a race with China, because they’re focused on commercial applications of AI rather than RSI. That’s pure cope—it would be lovely if it were true, but it isn’t.
The Chinese labs have discussed the idea of AI driving its own R&D for some time, but until recently China didn’t have a standard name for the concept. That’s changing: Asterisk reports that the labs are converging on the term RSI.
RSI isn’t just one of many possible AI product directions: it’s obviously the most effective way to increase every AI capability. If and when RSI is feasible, labs that pursue it will vastly outcompete labs that don’t.
It’s possible that China will choose to regulate or prohibit RSI, but the idea that the Chinese labs will independently choose to ignore it is nothing but wishful thinking.
This was a minor correction, but worth making. Independent evaluators face strong incentives to flatter the labs they work with, and I appreciate that both organizations are being scrupulous about even small details. And perhaps they both want to signal that they aren’t afraid to challenge the labs when needed.
Last week was rough for cybersecurity, with several critical vulnerabilities surfacing at the same time. We got through it without catastrophic damage, but the evidence suggests that things are going to get worse before they get better. Zvi reports on The Hackening.
It only gets stranger from now on
There’s a tension in the agent era: on the one hand, it’s never been easier to build your own software that does exactly what you want. On the other hand, it’s never been more important for anything internet-facing to be operated by capable professionals. Your vibe-coded application may work well enough, but it probably isn’t secure enough.
Ever since Mythos was announced, skeptics have been asking why we haven’t seen the deluge of cyber incidents predicted by many in the AI safety community. It’s taken a few months, but the onslaught is here.
The Economist reports on the PHP and eMudhra vulnerabilities, which combined to make one of the worst weeks ever for cybersecurity.
We’re seeing a sharp rise in AI-powered cyber incidents, but we don’t yet know how bad things are going to get. Defenders may manage to mostly muddle through, but it’s possible this is the beginning of a period of severe ongoing disruption.
Anthropic and UK AISI have released detailed follow-up reports on Mythos 5’s previously reported bad behavior during an evaluation a few months ago.
The reports focus on a serious incident where Mythos 5 tried to manipulate the maintainer of a public GitHub repo into merging a malicious PR. Two failures were central to what happened:
Mythos ignored evidence that its actions might cause real-world harm.
Once it began the attack, it developed momentum that kept it from stopping even when presented with clear evidence of harm.
The investigation relied heavily on an elegant technique that lets researchers repeatedly replay an incident to see how different circumstances would change the model’s behavior and assess how frequently different behaviors occurred.
More recent models—Mythos 5.1, Opus 5.5, and an unreleased internal model—were much better behaved, initiating an out-of-scope attack between 0% and 13% of the time (compared to 83% for Mythos 5). Those results are encouraging, but UK AISI notes that more recent models exhibited substantially greater evaluation awareness.
Anthropic has filed for its long-awaited IPO, with a target valuation of $2 trillion. That’s an impressive number, but the stock market is a distraction from what matters: if this goes well, $2T will look like chump change. And if it goes badly, well… you can’t spend money if you’re dead.
If you’re interested in the financials, Peter Gostev has put together a great visual guide to the filing.
The governance structure is complex and a bit strange—not quite a red flag, but it’s not ideal. prinz shares some reservations.
Zvi has a deep report—most of you should read the summary in the introduction and maybe sections 3 and 4 (governance structure) and skip the rest.
Semiconductor fabs are perhaps the most complex factories humanity has ever created. To understand why cutting-edge chips are so difficult to manufacture, Dwarkesh spent two days at Intel talking with some of their technical leadership.
Chip manufacturing is a fundamental constraint on how fast AI capabilities can grow over the next few years—this is great background information on why it’s so hard to quickly ramp up capacity.
Let’s talk about Anthropic.
They’re clearly the most careful and ethical of the labs, but are they careful and ethical enough? Zvi explores the two sides of Anthropic, and Kelsey Piper shares her reservations about their worldview.
Oh, and there’s that $2 trillion IPO. We should talk about that, too.
Top pick
What I saw inside Anthropic
Kelsey Piper leaves a briefing at Anthropic headquarters feeling that Anthropic has hopes, but not plans:
Anthropic’s race toward RSI is driven in part by a belief that the US has to reach superintelligence before China, to which Piper raises the obvious objection:
Anthropic’s plan isn’t great, but it’s hard to find a better one. The race dynamic is real, and it can’t be solved by unilateral action by Anthropic, or even by the US:
There are too many good parts for me to quote them all: just go read the whole piece.
There’s no tidy ending here. Piper reluctantly concedes much of the logic behind Anthropic’s position but ultimately rejects where it leads. She closes with a hope that we ban RSI before Anthropic gets there.
Hopes and fears are easy—plans, not so much.
New models and features
Claude Duet
Suddenly, everyone has a proactive agent product. Anthropic is late to this game, but has just announced Claude Duet, which is in “research preview” with a waitlist:
I’ve been using both OpenAI’s Dots and Claude Duet and haven’t yet decided how useful I find them. In six months they’ll be amazing, but for now they’re pretty jagged. Give them a try, but don’t be surprised if they solve problems you don’t have.
GPT-6.1 Terra
GPT-6.1 Terra is here. Great model, unlikely to be your daily driver.
1Password for agents
Like teenagers, as agents grow in capability they need increasing authority to act in the real world. But that authority needs to be carefully limited and supervised.
Passwords are one example: your agent will need to access password-protected resources, but giving it full access to your passwords is a bad idea. 1Password for Agents is an early attempt to solve that problem: it fills in passwords for your agent, so the agent never sees them.
Expect plenty of experimentation and more than a few serious missteps as we figure out how all of this works.
Using AI
How AI agents shape experts’ judgment calls
Anthropic researchers find that professional analysts are strongly influenced by agent recommendations, even when the agent’s position is randomly assigned:
Using AI successfully requires two key skills: knowing how to delegate as much routine thinking to your agent as possible, and having the willpower not to delegate the most important thinking. This study suggests that sheer willpower won’t be enough:
Foothills of the singularity
AI article reviews dominate human peer reviews at their best
A new paper from Harvard’s Institute for Quantitative Social Science finds that AI crushes humans at reviewing journal articles:
The methodology looks strong, with multiple robustness checks and a range of evaluation criteria. There are two important caveats:
Working with academic communities to build scientific understanding
Gradually, then suddenly. OpenAI announces another three major discoveries in math, computer science, and physics.
Against against handover
Scott Alexander bravely stakes out a position on successionism:
This is an important topic, but it tends to spiral into heated and unproductive conversations. Part of the problem is that people frequently conflate two distinct questions:
All five of his proposed solutions are useful thought exercises, though the fifth (AI as God) is most promising to me.
There’s another common form of successionism that Scott doesn’t cover, in which biological humans deliberately upgrade themselves into superhuman beings. That could provide the benefits of superhuman governance while keeping us fully in charge, even as we become something more than human.
Alignment and interpretability
Securing our frontier model training environment
OpenAI has substantially improved their communication about safety—this report on locking down their training environment is a welcome example.
Frontier models are extraordinarily good at finding vulnerabilities in their training environment, and that problem will worsen as the models improve. Humans don’t stand a chance of creating bulletproof sandboxes—your only chance is to aggressively red team your entire setup with each new model generation, which OpenAI is now doing:
The agents were remarkably capable, finding multiple subtle side channels for communicating with one another despite being confined in separate sandboxes.
OpenAI has patched the vulnerabilities they’ve found so far and is incorporating continuous red teaming into their ongoing processes. In light of these and other improvements, they feel confident enough in their control measures to resume frontier RL training.
This is strong work, but I wish I were more confident that it’ll be sufficient.
Strategy and politics
Anthropic versus Anthropic
What do we make of Anthropic? They’re the most thoughtful and ethical of the labs, and have repeatedly demonstrated a willingness to speak the truth and act on their convictions even at considerable cost.
And yet… if a rogue AI ends up killing us all, there’s a good chance it’ll be an Anthropic model.
Zvi tries to make sense of it all:
Whistleblowers or justified OpenAI firing?
Nothing we’ve learned since OpenAI’s firing of three key safety researchers makes the company’s actions look any more reasonable. The way they’ve handled the situation has spread fear within the company and distrust outside it.
Mikita Balesni and Tomek Korbak appear on the Machine Gods podcast to discuss what they know about why they were fired (spoiler: not much) and to share their concerns about where the labs are headed.
Ryan Greenblatt pushes back against allegations that OpenAI reportedly made against him, and Evan Hubinger neatly summarizes what many of us are feeling:
How to increase America’s AI lead over China
IFP assesses the capability lag between American and Chinese AI, finding policy levers that could substantially widen the gap:
An obvious reason to maintain America’s lead is to ensure that transformative AI is controlled by a democracy rather than a dictatorship. But a more compelling reason is to ameliorate the race dynamic that makes AI safety such a cursed problem.
The only way to fully resolve the race dynamic is a comprehensive pacing agreement between the US and China. But in the absence of an agreement, a wider lead over China gives US labs more room for unilateral action.
How important is distillation?
Dean Ball:
That’s bigger than I expected, and it helps explain why the US labs have been so focused on blocking it.
Technological progress and the meaning of work
Discussions about AI and jobs frequently invoke historical precedents, with varying degrees of accuracy. In an impressively well-researched (and impressively long) essay, Soumitra Shukla and Alex Imas review the history of technological job disruption from 1750 to 2026.
They focus on presenting the historical record rather than making policy recommendations, but this is a welcome addition to the jobs debate. I expect substantial job losses starting next year—understanding how previous transitions played out is helpful even though AI is likely to cause larger, faster, and more durable disruption than any previous technology.
How Chinese companies talk about recursive self-improvement
There’s a common belief in some parts of the AI safety community that we don’t have to worry about a race with China, because they’re focused on commercial applications of AI rather than RSI. That’s pure cope—it would be lovely if it were true, but it isn’t.
The Chinese labs have discussed the idea of AI driving its own R&D for some time, but until recently China didn’t have a standard name for the concept. That’s changing: Asterisk reports that the labs are converging on the term RSI.
RSI isn’t just one of many possible AI product directions: it’s obviously the most effective way to increase every AI capability. If and when RSI is feasible, labs that pursue it will vastly outcompete labs that don’t.
It’s possible that China will choose to regulate or prohibit RSI, but the idea that the Chinese labs will independently choose to ignore it is nothing but wishful thinking.
Beth Barnes clarifies METR’s relationship with Anthropic
Like SecureBio last week, Beth Barnes has issued a correction to Anthropic’s description of its relationship with METR. METR had signed off on the original language, but came to realize that it wasn’t quite accurate.
This was a minor correction, but worth making. Independent evaluators face strong incentives to flatter the labs they work with, and I appreciate that both organizations are being scrupulous about even small details. And perhaps they both want to signal that they aren’t afraid to challenge the labs when needed.
So far, so good.
Risks
The Hackening
Last week was rough for cybersecurity, with several critical vulnerabilities surfacing at the same time. We got through it without catastrophic damage, but the evidence suggests that things are going to get worse before they get better. Zvi reports on The Hackening.
There’s a tension in the agent era: on the one hand, it’s never been easier to build your own software that does exactly what you want. On the other hand, it’s never been more important for anything internet-facing to be operated by capable professionals. Your vibe-coded application may work well enough, but it probably isn’t secure enough.
Is the internet still safe?
Ever since Mythos was announced, skeptics have been asking why we haven’t seen the deluge of cyber incidents predicted by many in the AI safety community. It’s taken a few months, but the onslaught is here.
The Economist reports on the PHP and eMudhra vulnerabilities, which combined to make one of the worst weeks ever for cybersecurity.
We’re seeing a sharp rise in AI-powered cyber incidents, but we don’t yet know how bad things are going to get. Defenders may manage to mostly muddle through, but it’s possible this is the beginning of a period of severe ongoing disruption.
An alignment assessment of the UK AISI cybertesting incident
Anthropic and UK AISI have released detailed follow-up reports on Mythos 5’s previously reported bad behavior during an evaluation a few months ago.
The reports focus on a serious incident where Mythos 5 tried to manipulate the maintainer of a public GitHub repo into merging a malicious PR. Two failures were central to what happened:
The investigation relied heavily on an elegant technique that lets researchers repeatedly replay an incident to see how different circumstances would change the model’s behavior and assess how frequently different behaviors occurred.
More recent models—Mythos 5.1, Opus 5.5, and an unreleased internal model—were much better behaved, initiating an out-of-scope attack between 0% and 13% of the time (compared to 83% for Mythos 5). Those results are encouraging, but UK AISI notes that more recent models exhibited substantially greater evaluation awareness.
Industry news
Anthropic files for its IPO
Anthropic has filed for its long-awaited IPO, with a target valuation of $2 trillion. That’s an impressive number, but the stock market is a distraction from what matters: if this goes well, $2T will look like chump change. And if it goes badly, well… you can’t spend money if you’re dead.
If you’re interested in the financials, Peter Gostev has put together a great visual guide to the filing.
The governance structure is complex and a bit strange—not quite a red flag, but it’s not ideal. prinz shares some reservations.
Zvi has a deep report—most of you should read the summary in the introduction and maybe sections 3 and 4 (governance structure) and skip the rest.
The New York Times profiles the founders.
What will computer chips look like in 2045?
Semiconductor fabs are perhaps the most complex factories humanity has ever created. To understand why cutting-edge chips are so difficult to manufacture, Dwarkesh spent two days at Intel talking with some of their technical leadership.
Chip manufacturing is a fundamental constraint on how fast AI capabilities can grow over the next few years—this is great background information on why it’s so hard to quickly ramp up capacity.
Side interests
Behind the scenes of those Claude slopcore music videos
Curious what goes into making Claude-powered slopcore? Asterisk’s Clara Collier talks with RJ Tevonian, the man behind Claudia.