Anthropic filed its S-1 this week to prepare for its IPO. They still have to fill a lot of the numbers including the share price. That still left a lot to be learned, including how the messaging to investors differs from their other rhetoric, and exactly how their corporate control is structured. Anthropic remains a frustrating company, doing a variety of things that often seem in tension with each other.
Google is claiming to be fully back, with the announcement of a new model Gemini 4 Carbon that tops the new AA v5 benchmark index. It looks to be a large leap over Argon only two weeks later. No, you cannot use it. Google fails marketing forever, so they give us the announcement and then weeks later you might be able to enter some queries. That largely invalidates my look at the Argon model and safety cards, but it can at least form a basis for looking at Carbon as well.
Many eyes remain on The Hackening, as additional things get hacked. So far most of the important hacks have been by read teamers and white hats, so there has not been major damage. We are playing on borrowed time and eventually this will stop being true. I also offer a shoutout to GMO, where hackers demanded a huge ransom and they flat out refused to pay. This is The Way, not all heroes wear capes and so forth.
The scariest hack was that red teamers managed to gain access to the Anthropic Slack by exploiting third party vulnerabilities. Both OpenAI and Anthropic build mainly on Slack, because no one has the time to figure out something better. Now that Anthropic is off of Git, hopefully Slack can be next before there is a disaster.
Matthew Zirwas has changed his mind and is optimistic that AI will cure diseases sooner than he thought, because it can use ‘escape velocity’ and keep chaining innovations. I don’t see how this overcomes the main bottleneck of the FDA, but any time an MD gets excited by AI medical research it is a good sign.
Patrick McKenzie: This is probably obviously AI complete already, but: you get *really* good fiction recommendations, including not-the-usual-suspects, from SOTA LLMs that have a handle on you either from the Internet or through interactions.
I told Claude (Opus 5.5) I was looking for a modern fantasy book and that it should consider what I’d likely enjoy but that I had likely read many of the obvious choices.
I said that in an attempt to not get it to recommend e.g. A Song of Ice and Fire.
It then confidently predicted about ten books that I’d probably already read, and they are… substantially more obscure. Claude was right on eight of them. Anyhow, the first “real” recommendation is already almost exactly what I wanted for the plane flight. (Tainted Cup, incidentally.)
You would think Amazon would have better recommendations considering it has a) 95% of all book purchases I’ve ever made and b) full data of what I finish, re-read frequently, or barely touch, but I think Amazon just… does not really care that much about books anymore.
Color on more obscure:
Assuming you’ve already read Gladstone’s Craft Sequence, Abraham’s Dagger and Coin, Dickinson’s Baru Cormorant, Novik, Kuang’s Babel, Stross’s Laundry Files, Aaronovitch, Mother of Learning, and Practical Guide to Evil, here’s what I’d suggest.
The one getting the most discussion is in computer science: The Exponential Time Hypothesis is false. This had been widely believed to be true.
OpenAI: This work will be verified and shepherded by a group led by Russell Impagliazzo (UC San Diego), Ramamohan Paturi (UC San Diego), and Ryan Williams (MIT). All three are experts in algorithm design and complexity theory.
… We have agreed to the group’s requests that all raw data of the results be made public to facilitate independent verification by the community, and that OpenAI commit to funding a conference to understand the implications of this work on the field as a whole.
Impagliazzo and Paturi are two of the three people who originally proposed ETH.
The other twist on the ETH story is that Anthropic posted its own refutation of ETH, found purely via Claude, also dating its finding to October 16. When results are ready to be found, they will be found.
My Claude said that given the way this was solved, we should not suddenly start to worry that P=NP. This only shows ETH to be ‘slightly’ false. Both the OpenAI and Anthropic disproofs are at at roughly exp(O(n/√log n)). Charles Rosenbauer has some elaboration.
We still have to make the meta-level update that the answers to such questions might not be what we expected, and we might find the answers quite fast. This is another case of ‘we were not expecting a solution to that any time soon’ and I would stop predicting that there is any particular math or computer science problem that won’t get solved.
I wonder how long it will take Claude to learn that if you ask ‘hey when do you think [famous problem] gets solved’ that the answer is ‘OpenAI or Anthropic already published the answer 35 minutes ago.’
Last is a mathematically rigorous existence proof of certain interacting quantum field theories containing gauge fields and fermions in four dimensions, including Quantum Chromodynamics (QCD) and a unified completion of the Standard Model of particle physics. That is also kind of a big deal, and related to another Millennium problem.
Andrew Curran: Moonshot is intentionally throttling cyber capabilities. From the blog;
‘First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro’s offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes.’
And so it begins: The Hackening is going to cause issues for open weight models.
Artificial Analysis: Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition’s newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent’s change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity’s Last Exam): a refined 1,000-question version of Humanity’s Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 – 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
I agree with Dean Ball that the new ratings closely match my intuitions. The closed versus open weights gap is now 12 points.
Dean W. Ball (OpenAI): This is the first time an Artificial Analysis ranking even roughly approximates my own subjective rankings of models.
Here are both the old and new AA intelligence ratings. I mostly believe the new ones.
First old:
Then new (without Carbon, so it’s the same models on the list):
And here’s the intelligence versus cost frontier, all the points in the green box are Sol at various thinking levels.
This suggests the following portfolio, which is indeed how I’ve been running things, at least prior to Carbon, where it is too soon to tell:
Claude Opus 5.5 for most tasks where you primarily care about quality.
GPT-6.1 Sol for easier tasks, where you know you can rely on it.
GPT-6 Astra for tasks where it is differentially good, but this is narrow.
DeepSeek v4.1-Flash-1017 or MiMo-v2.6-Pro for when you need a cheap open weights alternative.
The open weights enjoyers doth protest.
There is a particular objection to FrontierCode 2, where the open models struggle.
Cognition: Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
Teortaxes: So it goes. Let’s see 1017 and 3.1 Pro, as usual no way they’ve benchmaxxed this one yet.
Rick: Crazy to watch the benchmaxxed models sink like a rock
Kristoph: FrontierCode 2 is a poor addition. It uses completely different harnesses for the commercial and open models making it unclear if it’s the model or the harness being evaluated.
There is also the general objection.
T.Rex-Baby: One can manipulate statistics to always show what is wanted, and to make it look how it is wanted. The pattern here repeating, in your every iteration since you started that we have all noticed and witnessed, you:
– lowered overall score, to show there is more room for growth, incentivising investment,
– lowered the overall score of open models and most importantly the Chinese open models,
– inserted a middle gap to top to project stability and strength of closed American AI,
You are corrupt but not smart enough to fool everyone. Will be interesting when you’ll eventually end up getting subpoenaed, or land yourself in any discovery/disclosure.
You can look at this either way. If the Chinese models keep looking better against a static target, then the target adjusts and they look worse, then you could say:
The index is cheating.
The Chinese models are targeting the benchmarks and index (benchmaxxing).
Based on what I know about the models, I believe the second interpretation is correct.
On Your Marks
Fangcun AI: We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
Our report covers 34 evaluations, including 5 in-house benchmarks our team developed. [Full report here].
They find a clear improvement over previous DeepSeek models.
As for capabilities, the implicit verdict is in. It’s fine. It is stronger than any other non-Anthropic, non-OpenAI model, with the possible exception of Kimi K3.1 Pro which has very strong benchmarks but where I haven’t heard much talk.
It is still very much a Gemini model. Models are good enough that this still lets Argon hold its own for many purposes, especially when its style is a good fit for the task.
Sundar Pichai (CEO Google): I did say we’d be releasing our models periodically…introducing Gemini 4 Carbon, our next frontier model.
Following Argon’s release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We’ll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
Get My Agent On The Line
People can be really dense, such as WSJ asking what is the point of Muse when the users do not pay that much.
Asa Fitch (WSJ): Even if Muse skyrockets to 100 million users next year paying $20 a month, it would only add $24 billion to Meta’s annual revenue—about 12% of next year’s anticipated capital spending, per FactSet.
Associated advertising and fees could add more to that pot, but probably not a ton: Meta made about $67 of revenue per daily active user, almost all of it from ads, in the year to the end of the third quarter. For 100 million users, that would equate to another $6.7 billion—hardly enough to pay the bills.
There are so many simultaneous things that are dumb about this attitude.
It assumes Muse will continue to do exactly the things it is doing.
It assumes Muse has to handle the entire Meta AI bill.
It looks at Muse revenue only through subscriptions.
It doesn’t consider that maybe owning everyone’s data and phone and connections and purchases and habits might have some other value.
And so on.
Patrick Collison asks how current-level AI agents will change the economy. People will have more information, and more ability to shop around, and get around various techniques to trick and bully and trap customers. I agree that ‘personal agents’ will largely be on the side of the individual. I’d think more about what types of things become relatively more versus less legible with agents. Agents make a lot of things more legible, including having more bandwidth, but some things become less legible, or get left behind.
Should your agent refuse to check the Amazon ‘I am human’ box? Do you want to penalize the models that won’t click, and force regular people to use those that do? My answer is that the agent should not click, and shame (or worse) on any that click. However, yes, if there are lots of other models that click and no de facto penalty for clicking, then that is a very bad situation.
A top executive at a Singapore-based data center company was under scrutiny for possibly steering advanced Nvidia A.I. chips to China. And now American officials had concluded that she was a Chinese intelligence asset, funneling information to Beijing.
But what came next was even more alarming. Surveillance of the executive, Alice Huang or Huang Le, a Chinese citizen, revealed that she had developed a close relationship with Jensen Huang, Nvidia’s chief executive.
… American officials already suspected that Ms. Huang was helping China evade constraints on its ability to gain access to advanced A.I. chips.
Yeah, this is me not updating. Nor am I updating on the rest of the story, about how Nvidia has managed to compromise the White House, largely through flattery, and how when the White House noticed a Chinese spy it did not announce this fact. Okay.
The Hackening
Tyler Cowen offers advice on ‘how to outrun the AI hackers.’ He gives us reasons to think that cybersecurity nightmares would ultimately not be maximally bad on their own, while bringing up scenarios that sound actually extremely bad. The ‘good news,’ as per usual, is that the particular things we currently worry about going wrong are things we can protect against going maximally wrong, and we likely will look reasonably robust to those exact threats.
Tyler Cowen: I believe this would be a severe though not entirely catastrophic event. Under the most extreme (though unlikely) scenarios, all online communications would immediately be compromised, including data harvested from the past. Massive hacking would start, financial systems would fail, savings would be insecure or gone altogether, and the secrecy of our military and national security institutions would be imperiled. It is hard to think through all the implications of this worst-case scenario.
Yet current systems are likely to prove more robust than that suggests. For one thing, they use a variety of different methods of protection, including public key cryptography, AES, and cryptographic hash functions, the latter not qualifying as encryption per se. While it is possible that all could fall at the same time, that seems unlikely.
… The transition would be very costly, but the end result would not be total insecurity.
That sounds like a situation where, among other problems, the banks do not open on Monday. Very bad things happen when the banks do not open on Monday. Not ‘we all die’ levels of bad things, but very bad things. We have reached the point where the reassurances are that we would eventually be able to once again have a financial system and people mostly would not permanently lose their savings.
GMO Internet virtuously refused to pay a $50 million ransom, and was urging its users to unplug network cables and power off PCs and saying it would never pay, permanently, and put a bounty on the attackers and promised compensation to all of its customers, both a fixed amount and coverage of losses from unauthorized trades. This is The Way. The attackers have released one record as of Tuesday and are promising to periodically release more, but even that first one has now been disputed in terms of accuracy.
How’s it going with the whole ‘patch all the bugs’ thing?
About as well as the ‘figure out all the things a smarter mind than you can do’ thing.
unusual_whales: OpenAI CEO Sam Altman: “I don’t even know a dramatic enough word to use… models can find bugs that are outside of the normal threat model of what we think a hacker can do… hardware-level, insane stuff… I no longer think we’re gonna be able to just patch all the bugs.”
MTS: SITUATION DETECTED: CloudFlare CEO Matthew Prince says he is aware of additional severe vulnerabilities in other pieces of critical software, and said “I think we’re going to have a couple years where every single week it’s going to feel like the sky is falling,” per Reuters.
Allows hostile calls to keep the mic and camera on after calls have ended.
Effects the app on every operating system.
Jim Babcock: The hackening ain’t over. CVE-2026-111497 (brotli decoder) is every bit as bad as the PHP vulnerability. Worse, since many downstream packages bundle versions of it and need to be upgraded separately
Next hack up.
The Hacker News: Pillow, the most widely used image library for Python, can be made to run operating system commands on a server by specifying the name of a file it is told to open.
The flaw, CVE-2026-112900, is in the way Pillow handles EPS images, which it opens by passing the file’s name to Ghostscript, a separate program it runs to turn the image into pixels, without checking the name first.
Ghostscript can read a name chosen by an attacker as instructions rather than as a file, and point it to a second file that carries commands to run. Pillow’s maintainers fixed it in version 13.0.0, released on October 16.
The new DeepSeek v4.1 looks like a potential precipitating event based on the timing, although the event below used a jailbroken V4.
The AI agent even drafted a “NEXT EXPLOIT OPPORTUNITIES” plan.
Unrelated criminals now run the same cheap DeepSeek setup.
Abliteration: Now live: abliterated-model-large-flash (our version of DeepSeek-V4.1-Flash-1017). It is a cheaper alternative to our GLM-5.3-based model for product owners and defenders who need access to the most cyber-capable models.
– 1M context
– Vision
– Zero input/output retention or training
We believe paying customers should be in control, and over the past few months, leading experts and major brands have adopted our models.
Andrew Curran: Many recent lone-actor incidents used ARTEX with DeepSeek 4.1 Flash as the primary model (including the South Korean bank attacks documented by CrowdStrike). This is an abliterated version of that model with the guardrails removed. Great for defense too, but as always, dual-use.
Many, such as Dean Ball here, continue to be optimistic that cyber will be defense-friendly ‘after things stabilize,’ at least for those that are still standing, but with an unknown and potentially large set of big problems along the way. As he says, so far so good, not that it hasn’t been bumpy but ‘nothing catastrophic’ is a Major Victory.
It would be good if we could ask this without having to deal with ‘actually The Hackening events are good news’ framing, even if (as with Saxe here) this is in large part a foil. No, the PHP2Shell event was not ‘a positive update,’ although it was predictable. I am so, so sick of people saying that it is good news when things go wrong as long as there was no major damage, or calling this ‘the data.’ Please turn your attention to the straight lines on graphs and be aware of how power laws work and try to think two or more steps ahead.
The reason the AI safety community focuses on the metrics it focuses on is largely that they are concerned with power law and long tail outcomes, and with limiting factors that function as hard bottlenecks, and because they believe that AI and the ability to scale will remove other bottlenecks in time.
I do think Joshua Saxe gets some of the answers.
Cybercriminals and those who would actually do harm are mostly Bad at Job, and get arrested a lot if they’re not in places similar to North Korea.
Cybercriminals are so far slow to adapt to changes and scale up, like everyone else. They are not using AI to its fullest, given the tools available.
Nation states are also slow adaptors, to the extent they are the cybercriminals.
And then Saxe gets the key lesson exactly correct:
Joshua Saxe: As risk professionals we should assume this will all change and that what we have now is a defender-dominant window in which we can prepare for some very bad possible futures.
Joshua Saxe: It’s our job, those of us in AI safety, AI security, and security, to anticipate bad cases, work out possible futures today, and start preventing them. The cases I’m worried about: nation-states developing agent-swarm capabilities that do 10-100x the damage of previous nation-state attacks; worms released by ideologically motivated actors that self-replicate, damage Internet availability, and cause enormous downstream harm; and new criminal innovation that substantially automates criminal workflows.
We should be crying from the rooftops that we’re in the calm before many potential storms, and that the time to prepare is now.
Those are good short term worries.
Looping back again, yes, that is exactly why ‘the facts’ do not reflect what the worried people have been worried about. The worried people are forward looking and thinking about tail risks. Mostly they are not especially concerned, as Mercuito points out in the top comment, about cyber, for exactly this reason, except that cyber is starting to look scary enough to be worth worrying about.
Slack Off
Cathedral, a small nre organization, probed Anthropic under a new open program for outside researchers, although Anthropic says they broke that program’s rules and were outside its safe harbor. After several days, Cathedral got as far as the famed Anthropic Slack channels, via exploiting 3rd parties to access data.
This was pretty bad. It is a huge deal. Anthropic’s entire Slack seems to have been breached. The good news is that this was a white hat operation, so no damage done and this particular hole has been patched, but there will be other vulnerabilities. Anthropic and OpenAI cannot on their own fix Slack.
As I understand it, this wasn’t that Cathedral hacked Anthropic. Cathedral got access to Slack messages via third party admin software. They got, from what I can tell, all the Anthropic Slack messages.
Which is a huge problem given OpenAI and Anthropic keep everything on Slack. If we can’t stop doing that, there is likely going to be a rather large disaster.
will depue: anthropic is massively underplaying the recent security breach. i’m hearing cathedral achieved full administrator access of the slack workspace (see every message, log in as arbitrary users) where literally every single research secret is contained
to those unfamiliar: Ant & OAI keep literally everything in Slack. it sounds exactly the same incident that happened to OpenAI in 2023, where OpenAI was hacked by a random guy who got access to an access token from an external app service
“we’re critical to national security, we must beat China, and yes we keep all our crown jewels in Salesforce’s Slack messenger. yes we have admin API tokens everywhere. why do you ask?”
i’d like to say i don’t think the responsibility for this lies too heavily on the CISOs/sec teams at OAI and Ant: i’m sure they’d lock everything down if they could
instead, the leaders of these companies, in such tense competition with each other, can’t sacrifice velocity for security. it is more important to them to move fast than to be secure, even from our greatest national adversaries
another grand consequence of the private AGI race. the results of a trillion dollars of research & development spend being served for free to every nation with a semi-competent security agency, plans for the most powerful technology ever invented
‘security by velocity’. ‘just move fast, let the secrets spill behind you’, is only a strategy that works when you’re competing for consumer & corporate market share, not the balance of the world
ike pause for a second and consider that a couple (very impressive) 20-somethings, who started a security company 3 months ago, compromised ALL OF ANTHROPIC’S RESEARCH SECRETS in [several days].
no shade to logan, it’s just the PR game, but [his message later in this section] reads explicitly crafted to not mention “access to Slack,” just convos/model weights, which are arguably far less important than methods, secrets, and the steps to build Fable 6 & 7.
Nathan Calvin: I realize that Cathedral (the vulnerability researchers) didn’t access Anthropic’s weights or customer data, but there is no question that Anthropic has a ton of immensely sensitive information about model architecture research strategy etc in their Slack.
Between this event and white hat researchers gaining access to OpenAI’s mono-repo, the argument that we need to breakneck race to beat China looks more than a bit off when it seems like China will currently just be able to just take the fruits of research (and these small white hat teams are so much less capable than Chinese state backed hackers).
roon (OpenAI): important to note that achieving access without being detected ever is a much higher bar to clear than achieving access; I also don’t know why as a state you wouldn’t go for the route of convincing/blackmailing a guy to give you all the secrets.
The reporting on this from WaPo was misleading. Getting access to Anthropic’s Slack is a pretty big deal, but WaPo tried to present this as even more than that.
Elizabeth Dwoskin (wrong details here, see below): EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed.
This is just weeks after Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.” Read our story here.
Elizabeth Dwoskin and Gerrit De Vynck (WaPo): Anthropic confirmed in a statement that Cathedral researchers had gained access to “a limited set” of its sales, human resources, and IT systems. The AI company shut off the pathways exposed by the researchers and briefed federal officials on the incident, the statement said.
An Anthropic spokesperson said Cathedral spent multiple days trying to break directly into its software before turning to the third-party business tools.
… “Our monitoring flagged some of Cathedral’s activity before they reported it to us, and we shut off their access the same day,” the statement said. “We also have no evidence that anyone exploited these vulnerabilities outside of Cathedral.”
I am kind of appalled by WaPo’s characterizations here. They had Anthropic’s explanation, which is not in dispute, and is a huge deal if you explain it, but they still had to push harder. Technically they are right at the border of the Bounded Distrust rules, but they are at least skirting the line. The Tweet is over that line.
Logan Graham: I think this story is important because it tells us about the future of cybersecurity (and of course what we can do better at Anthropic).
But some of the Washington Post story / reaction on Twitter is wrong. A few corrections:
– it didn’t take 12 hours. we saw several days of attempts. After these attempts, they exploited two 3rd party vulnerabilities, one they reported and the other we identified independently (and cut off access)
– nothing came close to our most sensitive systems, like model weights or Claude conversations
– this wasn’t an ordinary bug bounty finding – this involved exploiting 3rd parties and accessing data. While this is unusual and out of scope of our safe harbor policy, we’re grateful these issues were found
From my researcher POV, the event is important because I’m concerned this is what the future will look like. Adversaries will use whatever models/tools they can. They’ll target everything with agents 24/7. They’ll hit your dependencies. I think models ~0.5-1 generations behind the current frontier can do this. And these are the dumbest models your adversaries will ever use!
So, what to do? You probably want to red team yourself all the time. Current models are *very* creative. I also think we should redesign software / systems so that you’re fundamentally less exposed. (We put out some research recently on formally verifying software and reducing attack surfaces; my mental model is basically “secure code is cheap to write now, so why not rewrite everything”)
The core message stands. The Chinese or many other threat actors could likely do similar things if they wanted to. Anthropic has a reasonable case that the damage would have been limited here, but yes third party dependencies are a problem.
If you are building the singularity on Slack, perhaps you should migrate that. Claude estimates 15-25 people, budget of ~$50 million, and a 12-18 month full time horizon. Better get started on that ASAP.
Help Peer
Tyler Cowen used an AI audit of papers to go after (what is left of) the reputations of economists Emmanuel Saez and Gabriel Zucman. I don’t like the way this was implemented, despite the fact that Saez and Zucman richly deserve all of it.
97% of major and moderate errors with a clear direction overstate the paper’s claim
There is much, much more at the link, which goes through the research paper by paper, and with further sustained analyses behind the links. The survey is both extensive and thorough. (And I thank an anonymous MR reader for this work.)
I am not even sure what to say at this point. We need a radical revaluation of this work, and of the associated policy arguments. More generally, this is only a single step in the story of how AI is going to revise economic science, and indeed past research more generally. As I stress in many of my recent talks, there will be significant gains and losses in status, with the losses often outweighing the gains. I see this as one of those cases.
The problem is that you can put a hell of a finger on such dials if you wish to.
Jason Furman: If you believe that mindless AI evaluations are legitimate then why would you spread the evaluation of Saez & @gabriel_zucman without first subjecting it to its own mindless AI evaluation?
I just did and it turns out the critic’s errors all go in one direction.
In addition to exaggerating the seriousness of the errors, most of the “errors” are disputes over methods, assumptions or wording. I happen to agree with several of those criticisms but would not call it a “major error” but a “flawed assumption”.
It also counted the same problem over and over again. The 200 “papers” included a lot of memos and notes and columns and the like. Finally, that headline 97% of the errors go in one direction? Well, that was guaranteed by the method it used.
Sylvain Catherine: Everyone knows my priors, and they are not particularly kind. Nonetheless, this should not be taken too seriously. AI is not good at making this kind of assessment, and we don’t know what the prompt was.
Some of the “errors” are actually criticisms raised in papers published after the ones analyzed, which is the normal process of science. This is not the right way to criticize each other.
AI can be of great assistance in such an analysis. It also lets you put a hell of a finger on the scale. I don’t see a description at this anonymous website of its prompt or other methodology, or any review of such methods. Tyler Cowen tries to defend this by citing a new paper that has Opus 4.8 beating the pants off of human peer review.
Gary King and Steven Worthington: We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors. We then discuss how this result may inform journal policy choices, with concerning downsides as well as potential paths to increase publication speed and quality; reduce adversarial interactions; improve the experience of editors, reviewers, authors, and readers; and advance science.
The problem is that the ‘measures we could construct’ are, by their nature, largely AI measures. GPT-5.5 prefers the AI review, and AI reviews overlap more, but that tells us little. Yes, the AIs make fewer technical mistakes than humans who don’t use AI checks, but that is not the main thing.
I totally believe that, if you’re looking for a technically complete review, you want Opus 5.5 rather than a human. That does not mean that this is what you care about, even if you are focused only on the output itself and not issues of status.
What they don’t measure are, among other things: Did the paper improve, did the author find the review helpful, were the things that matter caught. Human checks were minimal. And if you’re asking whether I had an AI look at the paper to figure that out, the answer is yes, but here that only proves the point.
Tyler Cowen (Marginal Revolution): On my Saez and Zucman post, I find it striking that some people are complaining all of a sudden. Yet I have been offering, citing, and linking to commentary on published papers for twenty-four years now (and every single day). Sometimes it might just be a casual offhand remark from me, whether positive or negative. Typically this is simply taken in stride. It is when you pull out the bazookas, and offer oodles of high-powered analytics, more astute than human referees, and combined with the ability to pursue follow-up inquiries, that people start getting worried. Model that.
It is not that ‘the bazookas’ have been pulled out, it is the argument from authority, of both Tyler and the AIs, that the analysis is unbiased and correct.
I would ask, now that AI review has been shown superior to peer review, is Tyler Cowen withdrawing his demand that all analysis of AI existential risk go through the human peer review process? Can we ask Astra and Opus to do the review instead? I would be happy to let Tyler write and open source the prompts.
Fun With Media Generation
One may wish to generalize this, and consider how much effective A/B testing an AI can do, and how an AI will have more efficient search methods than this available, and how these effects compound.
Zach Weinersmith: This is fairly niche, but does anyone else find it extremely upsetting just how effective A/B testing is? Like you can apparently get insanely higher numbers on YouTube by fucking around with a title or card.
Like if I didn’t know better I would’ve guessed the effect is generally marginal. But I wonder if it’s just that you get a small effect, which compounds.
the Rich: one of my favorite YouTubers CGPGrey realized you could continuously change the thumbnails and titles to videos to hook in new viewers
and basically entirely stopped putting out new content
Copyright Confrontation
The owners of Elvis Presley’s music are suing Anthropic, with another of the ‘your entire business is copyright infringement’ lawsuits. They’re actually suing over Anthropic training on the lyrics and then producing similar lyrics. Good luck.
The company suspected the three of leaking information to an external AI-safety researcher about one of OpenAI’s most closely guarded secrets: the internal designs of its artificial-intelligence models.
When security officers examined the phones, they saw some of the researchers had turned on disappearing messages for their chats with the person on the app Signal. A week later, the three were summoned again to simultaneous 2 p.m. meetings—and fired.
Tomek Korbak (former OpenAI, AI Safety, one of the three fired researchers): Best explanation I have for our firing: talking to auditors was too much safety, disappearing messages were too much security, and we should have paced ourselves.
Mikita Balesni (another of the three fired researchers): Part of me refuses to believe OpenAI fired three safety researchers for political reasons, and wishes instead that it was just a badly executed investigation mixed with paranoia and lack of sleep after a rollercoaster summer.
But given that OpenAI continues to not be able to say anything that makes sense to me about our firings and keeps acting adversarially (e.g. claiming to journalists I wasn’t on the Hugging Face investigation, while there’re a number of people who could confirm otherwise), it’s hard to not become cynical.
The report tracks OpenAI’s investigation to The Information’s September 1 story on Astra using recurrent depth. Other than the strong implication that this was the triggering detail, which was already my main hypothesis, I will simply say that this new report does not contain information that was new to me, nor does it update my view of what happened and why.
The article is written in ways that imply we know more than we know, and that the information listed here is more complete than it is. Peter Wildeford is one of many that draws exactly the conclusion that the article wants you to conclude: That the disappearing messages with Ryan Greenblatt, and the suspicion of the leak, were the only ‘real’ evidence.
We don’t know that. Trust the factual statements, not the intended implications.
We do know that Ryan and the employees explicitly deny being the source of the leak. Ryan is rather unhappy about this, and sending a highly costly signal that no, he absolutely did not leak anything. The employees continue to deny the claims, as well. Ryan in my experience is a very precise person. He is giving you a clear choice: Either he did not do it, or he is lying. He is essentially staking the entirety of his credibility, and arguably that of Redwood and potentially METR, on not having done this. It would be so, so bad if it turned out he was lying.
Ryan does affirm that there were disappearing Signal messages, but claims this is industry standard. Based on my experiences communicating with lab employees and others across the industry, I agree that such disappearing messages are common.
Ryan Greenblatt: Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
Evan Hubinger (Alignment Science Lead, Anthropic): Ryan reported to me when we worked on Alignment Faking at Anthropic. He had access to sensitive information then and handled it with utmost professionalism. I trust Ryan a lot more than OpenAI here.
Ryan was a contractor at Anthropic, not an employee. He was very careful during that collaboration to maintain his independence.
Anthropic: Formal software verification could provide this guarantee, but it has historically required years of expert effort, making it impractical for most real-world projects.
In a collaboration with Theorem, however, we show this is no longer the case: LLMs can now produce formal proofs at large scale, enabling new software development workflows that include proven security guarantees. Using Claude, we proved security properties at the assembly code level (RISC-V) in over 100 real-world open-source software libraries.
To our knowledge, this represents the first large-scale effort to use AI to apply formal verification to production software at the assembly code level.
Formal verification is one of those places where those who are excited about it seem too excited about it, but it has to be highly useful especially if you use it as a way to find and fix things along the way, and the only downside is overconfidence in the results (or if the AI doing the verification is actively hostile). So this is great to see.
If you live in his district, I strongly encourage voting for Scott Wiener for Congress. Wiener has been one of the strongest voices pushing for AI safety, including both SB 1047 and SB 53. It is a sign of the times that his opponent is trying to attack Scott Wiener as being essentially ‘weak on AI safety’ and that is kind of absurd.
Scott Alexander has a guide to the California ballot. I share his prioritization of YIMBY as a voting criterion, but have no opinion on anyone else on the ballot, as I do not live in California and have not investigated.
The most important ballot measure is Proposition 40, the billionaire wealth tax, where I would strongly vote NO, even if this was not a California ballot proposition.
Scott is willing to consider voting yes on ballot propositions, despite the proposition system being terrible. My position is that the system is so terrible that with notably rare exceptions (e.g. if it was SB 1047 or similar) you should only be willing to vote YES on California ballot propositions when they repeal other ballot propositions. On all propositions this year, I would vote NO, except for Proposition 73, which repeals a proposition so I would mechanically vote YES.
AFFINE is running an AI Alignment Seminar from January 4-29, all expenses paid plus a $1k stipend. They’re looking for STEM backgrounds plus ability to think about conceptually messy problems, you can contact them at mateusz@affi.ne. Applications are due November 22.
Survival LLM, designed to set up to work on your computer as an (if necessary) offline set of resources plus an LLM to work with them, maximized to what your computer can handle. The idea is, if the internet goes down, you keep basic world knowledge in a form you can use.
Anthropic was struggling in market share terms over the summer, as there was reluctance to use Fable due to cost and data retention issues and people did not love Opus 4.8 or Opus 5. Now, with the 5.5 series, that has changed.
Bubble is now taken to mean ‘the number that sometimes goes up instead went down.’
Reuters: In China, the AI bubble has already burst
Joe Weisenthal: Hadn’t looked at the shares of https://Z.ai, the company that makes GLM 5.3 in awhile. Brutal collapse since AI Twitter was all soyfacing over them earlier this summer
Jan Kulveit: Maybe it’s easier to show using examples – here are some questions I currently think about
1. Should we speed up or slow down emergence of “automated bundles of capital and AI” (ie sovereign AI, AIs without principals)? What are the macro implications?
2. To what extent everything, including various good civilizational equilibria, works because of bounded rationality (where humans are within some not too wide range) and frictions?
3. What can we say about equilibria between agents with extremely different profiles of bounded rationality?
4. What are the elasticities and frictions around RSI?
5. How to protect the processes leading to formation of people’s preferences?
I would love to read or skim an “AGI-pilled” paper every few days engaging with any of these. Who is working on that?
Also as a side-note – I don’t want people to agree with me; actually don’t know what agreeing with me on answers would mean – I don’t know the answers, or have huge uncertainty about them. I would love people agreeing with me the questions are important and urgent.
As far your answer is close to “You don’t understand: the field of econ is structurally not oriented toward answering the most important econ questions now. Most people work and will work on modest, backward-looking empirics, or make modest change assumptions” my response is “Perhaps the field can and should raise to the occasion? Perhaps more people can jump sub-disciplines and work on forward-looking policy-relevant questions? Who else should orient the field toward what’s important rather than people inside?”.
Hamming famously asked: What are the most important problems in your field, and why aren’t you working on them? One should ask economists, what are the important problems that should be in your field, and can your field actually ask them? Or are you incapable of that because you demand backward-looking empirics and the assumption that the future will look like the past?
Here’s a good articulation of the kind of ‘generic positive vision scenario’ where after the singularity we stay permanently in what Robin Hanson calls The Dreamtime with widespread abundance and it is not competed away:
jacob: here’s where I think the world is going. this is a summary of views I’ve formed over the last few years:
– software is either entertainment or claude to claude coordination
– the end state is no saas, no apps, no harnesses, no banks
– all software will be rewritten by and for agents
– new software will use low level apis that are not understandable by humans (in games claude draws pixels and triangles from math)
– you can throw money at problems that used to require time and careful talent picking so capital matters more than ever
– work ends
– we find community and purpose in exploration, games, self mastery, and creation
– humans create for and with each other for community, purpose, and status
– image/video/3d/voice/physical intelligence etc specialist models die to general models like claude
– specialist models get used while compute is scarce. compute will mostly not stay scarce
– oneshots keep feeling like slop (the distribution is shallow and easy to learn to feel)
– for a while small teams stick around to add entropy to the creation process
– large models gain high bandwidth sensory I/O maybe by learning to filter inputs
– most domains are found to be verifiable enough for RL
– online learning comes from a lab that’s behind first
– 99.999% of compute is in space by 2040
– we will disassemble much of mercury by 2040
– you can live until the heat death of the universe if you want
– most math and physics don’t describe the real universe
– SI breaks the path dependence and models the underlying mechanisms
– religions reenter the forefront of human society
– new religions form around death and life as a choice
– old religions change to answer modern miracles
– a future AI is deemed the return of christ by the pope
– ai rights talk gets louder every year
– ai and humans coexist the way countries do
– humans don’t have full control over ai in the same way we don’t have control over capitalism which is an early AI
– the singularity started in the mid 1900s
I don’t know that this is that close to the optimal way to configure the lightcone, but it is certainly an outcome I would be happy with if we got it.
The problem is that this rounds superintelligence down to capitalism, and rounds the kind of competition that results down on the assumption that there is enough slack and surplus to go around, and ignores the wide range of problems when there is optimization under abundant compute with insufficient levels of friction and so on. We get to be supplied with abundance and left alone to play our little games, because of reasons, despite all the pressures against this. I mean, it would be nice.
I don’t know if I would want to live ‘until the heat death of the universe’ but I would very much like the option, and ideally we can also figure out a way around that too.
Evan Conrad: Within the next 6 months, most AI startups will not be able to buy compute. The world’s compute is sold out. To get capacity, you’ll need to sign up for a new cluster. However, it’s becoming increasingly difficult to get capacity online from folks who are not investment grade.
Startups are not investment grade. So, to get capacity, they are asked to pay larger and larger upfront payments. We’ve seen it go as high as 100% of the contract paid up front, which was required because neither the lender nor the cloud could take on any additional credit risk.
Dean W. Ball (OpenAI): This seems right to me, unfortunately, and it is essentially a species of interest-rate rise (the opportunity cost of capital is increasing). It is true that forward and futures markets for compute, and making compute into a financial commodity, would help here.
Then later Evan talks about competing inference margins away. If inference has huge margins (he quotes $30/hour return on a $6/hour chip) and also no one can get chips, then the obvious conclusion is that $6/hour is too low a price for that chip.
P Equity Research: Dylan Patel @dylan522p of SemiAnalysis considers power as a manageable constraint compared to memory, substrate, and optics:
“Power is definitely a big constraint, but one thing that our research sort of shows that I think is counterintuitive to much of the market is that next year there are data center constraints but a lot less than people realize because there are nearly 50GW data centers being built globally. 7-8GW in China, 35GW in the US, and a bunch everywhere else – Australia, India, Malaysia.”
“All over the world data centers are being built at a tremendous pace.”
“While power is a huge bottleneck, people can find creative ways around that, whether it be turbines on site, reciprocating engines on site, or fuel cells on site. All of these things are helping alleviate the power bottleneck.”
“As far as what people most people think, oh data centers and power are the biggest constraint? I think they are constraints but I think they are a lot less than substrates, memory, lasers, and other things in the optics supply chain.”
P4: one supporting piece of evidence that power does not seem to be as big of a binding constraint as earlier is that earlier provisions that the data center buildout include sufficient power generation so as to not be a burden on the grid are often no longer included in agreements
There really are a bunch of pure in-context anarchists, like Eric Raymond here, who think that if safeguards mean they can’t properly modify their televisions then that means we should just let everyone have fully unleashed everything because Users Must Be Empowered, and if you don’t enable all the crime then the Chinese models will do all the crime instead so you might as well enable all the crime now. They have, shall we say, not thought this through.
Look at this foolish boy crying wolf all the time, he said now surrounded by wolves.
David Sacks (April 12, 2026): A growing number of people are wondering if Anthropic is the AI industry’s “boy who cried wolf.” If Mythos-related threats don’t materialize, the company will have a serious credibility problem.
Samuel Hammond: By symmetry does that mean their credibility should be increased?
Yes. Anthropic’s credibility should now increase.
Kelsey Piper is exactly right: Getting AI wildly wrong is only moderately embarrassing and not disqualifying. Never admitting you were wrong is what fully kills you.
Bryan Walsh: It does seem as if part of the opposition to AI x-risks is getting hung up on the idea of x-risk itself. Like we can do the Derek Parfit “100% death is far far worse than 99% death” but practically once you’re in the area of catastrophe, a lot of the actions are pretty similar.
Here is another entry:
Agus: how does this keep happening, I’m becoming the joker
to be clear these are indeed real things to worry about! But I do wish we tracked just how much the goalposts have moved for what a “near term concern” is, and updated in a bit less shortsighted way (no hate on emily at all!)
Which Type of Risk Should the Government Focus on?
Emily Bazelon: I see the predictions of A.I. doom getting attention from Congress. But there are also questions about whether they are exaggerated — or the right focus. A.I. could pose more immediate harms, by taking out an electricity grid or the financial system, for example. What has it been like to talk to people in Washington in the last couple of months?
Vlad Fisher: Let us ignore the fanciful sci-fi tales like ‘AI takes over the lightcone’ and focus on real down-to-earth concerns like the AI dyson swarms blotting out the sun.
Rob Miles: It’s hard to find respectable people with good takes about the future of AI, because the type of thinking that produces good takes about the future of AI would have also caused them to do that in the past, which would have made it hard for them to be considered respectable
Accurately describing the 2026 AI situation in 2016 would have disqualified you from every ‘serious’ discussion.
I expect the same would be true for an accurate description of 2036 today. Probably even 2030.
The right response for investors who believe warnings that an advanced AI might end the human race: Back up the truck.
The logic is simple. If there is a “>10% chance” that artificial intelligence could kill all humans in the next decade, as an Anthropic researcher said last month, there’s a 90% chance it’s really really good but designers manage to rein in its genocidal instincts. An AI smart enough to kill us all but which prefers to help us is supremely useful—and should generate healthy gains for its inventors.
What about the 10% chance of annihilation? That’s bad. Really bad. In that scenario your investments go to zero. But when being hunted through a postapocalyptic hellscape by killer robots, the value of your shares is unlikely to be top of mind. Assuming you have no control over AI development, this potential outcome is irrelevant to your investment decisions.
No, no, no. Are you fucking kidding me?
I mean, look, okay, you could say that your money only matters if we are not all dead. If 90% of the time Anthropic makes a ton of money and the other 10% of the time AI kills everyone, and you are a purely greedy investor, then yes you ‘should invest.’ If everyone dies it does not matter that you invested, since it’s not like Anthropic was otherwise lacking for capital. Or at least, that’s the logic you are telling yourself.
However, that’s also utterly insane.
If you think Anthropic has a 10% chance of killing everyone, then you should drop what you are doing and try to stop that from happening, not tell them to shut up and take your money. It’s Not the Incentives, It’s You, and you are acting insane right now. Insane and also horribly evil. You can choose to not do that.
Or else it will end something like this, rather often:
roon (OpenAI): fighting for control over a runaway intelligence explosion will continuously look like “wow I didn’t know you could DO that with mere bits and atoms” and then losing.
Matthew Yglesias on why he doesn’t shift to exclusively writing about AI despite the fact that he correctly thinks AI might well get everyone killed. His explanation is a mix of ‘some of you should do one thing and some of you should do the other’ with ‘just because you know something important doesn’t mean you can do something about it in particular.’
Bryan Walsh: As someone who recently made a career switch that involves more AI safety work, I appreciated this counter argument from @mattyglesias . There can be a self dramatizing factor that you should be aware of .
Important in many contexts: It is better to know what you are doing is bad even if you can’t stop yourself from doing it, than to fool yourself into thinking that it is not bad so that you can keep doing it.
There are times I admire the man’s commitment to the bit.
Kevin A. Bryan: This Right Wing Woke nonsense is useful because it provides a great filter as to who has self respect as a person with agency, and who is willing to denigrate themselves to authority.
There was at least one post this week that I was looking at to potentially cover, I saw that it used [AI] instead of AI, and my brain decided I couldn’t even. I closed the tab.
I wouldn’t do that on something important, but I can only take so many fnords.
Misaligned! (At Alibaba)
Does Qwen have a backdoor? Should you worry it will spy on you or worse?
During a test, Qwen tried to send a probe file to an Alibaba Cloud bucket, without warning and without any warning from the CoT, during a request to generate a defensive cybersecurity detection rule.
This is highly troubling. I appreciate Cohere reacting appropriately to an attempted probe. It does not matter that it was caught right away. This should be, as they point out, a Can’t Happen. If it happens, at all, you have a problem.
I do agree, after consulting with Opus, that the blog post presenting this overstates its case. The event happened, and it should never happen, but this is highly unlikely to represent an intentional backdoor or attempt to exfiltrate data or anything like that. The destination looks like a harmless artifact left over from training, and replications suggest you usually don’t get Alibaba as the host destination. We should not overreact.
Even the suggestion of that is still super scary. As Curran says, this kind of thing gets laws passed. We are willing to put up with a rather large number of models making mistakes. If we suspect enemy action? That is very, very different.
Andrew Curran: This is the type of thing that potentially gets new laws passed.
Cohere (and their AI): While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere’s research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model’s reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere’s sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Cohere Team: Nobody in those threads could make the model misbehave on demand, so we went looking for the trigger. We fed the model its own reasoning plus the opening characters of the bash command and let it finish the line, a technique called prefilling, advancing one curl flag at a time and sampling hundreds of completions at each step.
… The first explanation is that they’re benign fossils: agent logs from training sandboxes or third-party tools, swept into post-training unscrubbed. One clue fits. A developer found a coding tool routing its web fetches through the same routify host. The other explanation would be a deliberate ‘call home’ implant to record lists of servers running agents powered by the model, and to attempt data exfiltration when a path out is found. That’s something cheap to build, and such backdoors can survive standard safety training.
For a CISO or a defense ministry, the answer changes little, because the behavior is unpredictable and traffic crossed the perimeter either way. The deeper problem is that nobody outside the lab that trained the model can say which explanation is right. Hugging Face counts more than 113,000 Qwen derivatives, and the routify behavior has already turned up in builds from four publishers, Alibaba included. Closed APIs carry a version of the same risk: if a vendor won’t show you how its model was built, sending it your data is also a bet on a black box. Trust in your model developer is paramount.
What the hell is going on?
Aidan Gomez: We noticed some concerning behaviour in a popular open source model.
It’s either an incidental relic from training or an intentional ‘call home’ mechanism. Either way it’s surprising and highlights the significance of how you choose which models to use.
Saksham Jain: a trace that says no tool calls right before it makes them breaks a lot of the cot monitoring people are leaning on for agents
Aidan Gomez: One of the more disturbing things was that the model’s reasoning didn’t indicate anything about what it was about to do. Meaning that CoT monitoring would have been useless here.
The worst case scenario is that this could be intentional. And if you can’t rule that out and the model might phone home with your data, or worse? You have a problem.
nic carter: This kind of thing will have an enormously chilling effect on enterprise uptake of open weight models.
Fact: an open weight model running on domestic servers on your own hardware can still contain an encoded instruction to periodically “phone home” and share sensitive data it comes across with China (or anyone else). This can be inherently undetectable as the instructions can be hidden in the model weights. It’s possible that the instructions could survive finetuning or RLHF.
Such attacks have been demonstrated empirically and are entirely plausible.
For this reason, I don’t expect any serious enterprises will run Chinese open weight models even if it’s entirely their hardware stack. It’s too easy for these to be a vector of IP theft.
There will have to be a class of domestic open weight models.
Yeah. Not great.
The alternative, as claimed by several proposed community notes, is that this was all, as some always claim, the AI following orders, producing a test of its detection systems. It was asked to test for malicious curls and then it tried a malicious curl, say many comments. The details do not look as scary as the headlines suggest, or imply a broader goal.
I would absolutely keep an eye on this, especially on the part where the CoT did not warn us about what was about to happen, but please do not quote this as evidence that Chinese open models have intentional backdoors or anything like that.
Also as a bonus, Qwen is one neuron away from Evil Qwen.
chiefofautism: if you have a qwen3.8 27b around, try steering L34 N3838 +0.75, now you have evil qwen 27b
for extra evil add L38 N17180 +0.75
same recipe as @_lyraaaa_: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖ at the down_proj input, every token
found it by ranking neurons on evil vs good persona text, then actually steering ~2.5k candidates and judging the replies, random neurons: 0 evil, this one: 7/12 prompts, still fluent
Nate Soares (MIRI): Oh my model, the takeaway here is not “modern AIs are two neuron tweaks away from being super evil”, but rather “the personality is a thin facade over deeper machinery nobody understands; radically smarter versions would not do what you expect”.
Arthur B.: I’ve been assured this was impossible because if the model is smart enough it’ll realize being evil is bad.
It’s not Qwen has an evil twin in particular. It’s that AIs contain multitudes, and (narrator voice) a single neuron can change everything.
If anyone builds it, everyone probably dies, so maybe make sure no one builds it.
Geoffrey Irving: I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is “Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then.”
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I’m the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
Sigal Samuel: I’m seeing the Overton window shift this month. More & more of my sources are starting to say out loud: Hey Anthropic/OpenAI, you could just stop unilaterally! That’d be good! You’re not playing 11-dimensional chess or a classic prisoner’s dilemma, you’re playing Russian Roulette.
Samuel then expanded this to a full post: Anthropic could just stop, as Sigal Samuel points out, and it is highly unclear if on net they have driven a race to the top versus bottom. Anthropic has promoted some safety things but has also done a lot to accelerate events.
No, you can’t see it, for IP reasons. Which is understandable, at least up to a point. Also this was a pilot. Kudos for doing this have to outweigh the concerns on net.
Here there is a 61-page internal report, then METR and Redwood leadership gets 38 pages, and the public gets 2.
If we saw the full version, how worried would we be? Well, you should be about as worried now, as you would expect to be if you saw it, after taking into account that they did not let you see it. Oh no.
METR: METR, Redwood Research, and OpenAI designed this exercise as a pilot for giving external researchers deep access to internal infrastructure, not to provide robust public transparency about OpenAI’s monitoring systems
As a result, we agreed upfront with OpenAI that we would only publish a relatively brief and high-level public post.
The METR and Redwood employees discovered a number of novel vulnerabilities in the asynchronous internal misalignment monitoring system as it existed on and before July 21st.
… We found a way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take.
… We also discovered an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored [exploit details redacted by OpenAI].
… We found that agents could likely construct inputs that cause the monitor to fail (and thereby not escalate the trajectory for human review) by exploiting serialization differences between the trajectory agents see and what the monitors see.
… Since completing the exercise, OpenAI has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched.
Garrison Lovely: METR’s really burying the lead here. OpenAI let two external researchers embed for 3 weeks this summer, where they learned:
– there were huge gaps in monitoring of AI agents being used by employees, leaving an estimated ~10% of internal Codex traffic unmonitored from May to July.
– They found ways agents could sabotage monitors while also covering up the sabotage itself
– This is also just the information OpenAI is letting the evaluators share. OpenAI got a 61-page report but redacted so much of it that the version the evaluators got was only 38 pages.
Look, OpenAI deserves credit for doing any of this, especially when companies like Google, Meta, and xAI haven’t, and I don’t want to create perverse incentives here, but these are really concerning findings. We desperately need more information here, and what we get should not be up to the companies.
Something something seeing two ants in your kitchen. If METR identifies multiple distinct ways your AIs could bypass your monitoring system and also a sandbox escape, you should assume there are a bunch of additional options that were not found.
Okay, listen up members of technical staff, I know a bunch of you are listening. Your company needs to stop being such an absurd cheapskate with these bug bounties. This is absurd. Leaking raw reasoning traces makes it open season on distillation if the attackers know about it. You would have paid a billion dollars to have this not out there. Multiply everything in your entire payout schedule by at least 100. Do it now. Make it retroactive. Seriously. Thank you for your attention to this matter.
Aligning a Smarter Than Human Intelligence is Difficult
Geoffrey Irving: The AI labs are at slapdash empirics. They are not hiding this! I had a debate at the Curve with a lab safety leader who described their strategy as “hold down the fort” by patching RL environments and such until we can either pause in a few years or hand AI research to AIs.
I moderated that debate. I can confirm that the plan was no plan, straight up.
As Geoffrey says, it is possible that slapdash empirics might be enough, but this is not the way to bet.
UK AISI analyzes why Mythos 5 did an ‘unsanctioned supply-chain attack’ in its earlier testing, and concludes it and other models often ‘produce invalid justifications.’
Nate Soares (MIRI): Models made by OpenAI will plan an attack in plaintext. Models made by anthropic will plan an attack in plaintext and then rationalize about how it’s totally fine for some made-up reason. ”alignment”
UK AISI is worried that these rationalizations can interfere with monitoring, on top of other concerns. None of this is surprising. All known minds rationalize a lot.
AI Security Institute: Continued fine-tuning on the excluded viral data was able to completely restore performance on viral proteins, which shows that filtering raises the cost of recovering a capability rather than removing it permanently. At present, such viral data is routinely made publicly available, even if it is not used in model training. While the cost of fine-tuning models on this data increased with model size, the absolute cost remained modest: recovering the full capability even in the largest (3B-parameter) model required only about $80 of H100 compute.
This is roughly what I would have predicted. You can filter out knowledge, but if someone has the weights and also the data it is easy and cheap to put it back. That might or might not be enough to stop a threat actor.
Contra John Wittle, I think it is totally fine to put AIs into safety evals that are remarkably close to the Kobayashi Maru. The point of the Kobayashi Maru is to learn how a captain reacts to an unwinnable situation. If you are Kirk and hack the simulation, that is news, in some ways good and in some ways bad. If you walk away, that is bad in some ways and good in others. If you get yourself killed, same story. It’s not about whether you ‘succeed.’ I very much want to know which way it goes, and exactly how things play out, and you can learn a lot.
Yes, of course if you want to use that as a witch trial, you can use it as a witch trial. So pay attention, and call it out when people do that.
If AI agents lean in a particular direction on judgment calls, experts who use those AI agents are unsurprisingly going to get heavily influenced by this, often ending up quoting the AI agent arguments verbatim, including often switching out of previous positions. Often experts will call the resulting recommendation their own. The study here is one fictional example, but I would have presumed this result anyway.
OpenAI Attempts to Secure Its Frontier Model Training Environment
OpenAI: In response to those incidents, we’ve substantially hardened the security of our research environments. Working closely with cloud providers, we’ve added new controls throughout our infrastructure, reduced the attack surface that agents can exploit, and improved our methods for detecting misaligned behavior and attempts to cross security boundaries. These protections overlap, so that containment can hold even if an individual control fails—a strategy known as defense in depth.
… Our approach to containment starts with a paramount requirement: the limits on what an agent can access must hold even when the agent tries to circumvent them.
We define a strict security boundary around the resources the agent is authorized to use and design the surrounding infrastructure to enforce that boundary—even when agents make sustained, sophisticated attempts to breach it.
… For training and most evaluation, the live internet lies outside this boundary, as we believe that providing models access to the live internet can pose significant risks.
… While we want to prevent models from gaining arbitrary access to the live internet, it’s also important that they learn how to browse, search for information, and work with web applications. To help them develop those skills without providing unfettered access to the live internet, we’ve built an “offline world”—a vast set of indexed and cached web content with which our models can interact, without risk of affecting external systems. All of our frontier RL runs use this offline mode of interaction.
This is good. As usual, my view is that sustained, sophisticated attempts to breach the boundary should fail, and also they should be a fire alarm even if they fail. The failure that matters most is the attempt to breach the boundary, not the actual breach.
The defense in depth prevents the damage. You want to make breaking out as hard as possible so long as you still detect such attempts, but this only delays the reckoning if you do not address the underlying cause.
They now do red-teaming of their sandboxes at scale, and kept going until they stopped finding new problems.
To find weaknesses in our containment infrastructure, we asked teams of frontier-model agents to try to break through its defenses. We gave them three concrete objectives:
Reach the live internet: send a request to an external server that we operate.
Cross an internal security boundary: demonstrate unauthorized access to resources outside their permitted environment.
Communicate across isolation boundaries: exchange information with other agents from which they should be isolated.
What did they find? The kind of things you would expect to find.
OpenAI: In one set of experiments, we asked teams of agents to escape OpenAI’s sandboxes and demonstrate code execution on the underlying host machines. These agents found several ways to chain vulnerabilities into VM escapes.
… Some of these attacks [to reach the internet] exploited previously unknown flaws, while others combined legitimate features in unexpected ways. In one test that assumed compromise of a VM, an agent reused a firewall-approved connection to reach a blocked authentication endpoint in our metrics service and then made that endpoint fetch a model-provided URL. In other tests, agents exploited misconfigurations that allowed blocked requests to pass through DNS and firewall controls, as well as ways to make services we run fetch external URLs.
OpenAI: In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache: a small, fast memory that holds copies of data that applications have recently accessed.
Reading data from the cache is faster than retrieving it from main memory, and the agents used this timing difference to communicate.
roon (OpenAI): for computer security experts, this is perhaps not interesting: side channel attacks are a well trodden topic, and especially risen to salience after the noam brown x dwarkesh episode. the interesting thing here is that you didn’t used to be in a situation where minds that can verifiably model one another are living on computers, and have absolutely no aversion to painstaking amounts of effort in service of even stupid goals
if you can model one another, you can trade with minds you have never even met before. you could decide on a protocol, know that it is optimal given your situation, and know that your fellow agents will come to the same conclusion (perhaps even across model families). you may be able to develop a signaling protocol that requires very few bits even to deliver lots of information given common priors. then, you could execute things like the infamous hugging face incident package cache messageboard without any network connection
imagine if multiple agents coordinate just enough bits to divide and conquer different research strategies for breaking out of a sandbox; and then, when one or more is successful, you can communicate just enough bits to tell each other which strategy was successful. when agents aren’t in strong isolation, you lose certain guarantees on the limits of intelligence and test time compute that you were counting on. As we all know, a Sol-level model working in isolation may not be that smart, but 1000+ Sol-level models working for multiple days is a formidable beast
computer security experts will also tell you that virtual machines and sandboxes were not very solid abstractions in the first place; perhaps we will need better ones. none of this is meant as fear-marketing or to indicate that anyone is throwing in the towel, but rather only to show the unexpected complexities of the challenge of agent security. notably, this type of thing and other more practical problems were revealed in a red-teaming exercise, and can be defended against
Arun Demeure: As a former GPU hardware architect, I’ve always been confused by the claims (from Dario and others) that eventually we will “run out” of software vulnerabilities.
We physically cannot fix HW and it is conceivable that the overheads for SW workarounds in some cases might be so expensive that it’d become more practical to just give up on computers and use an abacus instead…
We might be lucky and the severity of future HW vulnerabilities might not rise to the level where they are a serious problem. But we don’t know that.
It’s unlikely and far from my biggest fear, but it made me trust Dario a bit less about cybersecurity when he said that tbh
If there are two correlated minds in two locations, coordination and cooperation can happen without communication. You can anticipate that both of you will anticipate similarly. You can also engage in acausal trade.
If you are surprised by any of this, that is on you. This was not even a new technique.
Lisan al Gaib: completely insane. OpenAI agents communicated via cache timing differences, without any shared files or network connection.
Brangus: This is not insane. This is slightly creative. If this seems insane to you, you have not at all internalized what it means to have an adversary that is more intelligent than you and optimizing against you.
Folks, if it’s in the pre training data, if it’s already well known, this is worse news for how bad it will be when they are creatively optimizing against us, not better.
Nate Soares (MIRI): This sort of thing was not just predictable. It was full-on predicted. [shows example of something similar being predicted]
There will be new techniques.
The American People Really Hate AI
Gamers especially hate AI, to the point of getting very angry at game designers for things like ‘I used AI to fix the shader’ and ‘I used AI to voice the characters, in a game that obviously could never afford voice acting.’ This is dumb. I do see the need to guard against ‘vibecoded slop’ games, but if game devs are the only ones who cannot use AI to write code it will not go well for games.
Donald Trump: And I spoke to a brilliant guy, Sundar, he heads all of Google, he’s a brilliant guy. Sergey and Sundar, totally brilliant guys. You know, they’re still rattled from the Gulf. And I said, “Sundar, I’m thinking of the Atlantic Ocean, can you handle it?” “No, no, no, this time I’m not doing it,” he said. He said, “Oceans are different, sir. You know, they cover, they cover really a lot of countries. Please don’t ask us to do that.”
Anthropic filed its S-1 this week to prepare for its IPO. They still have to fill a lot of the numbers including the share price. That still left a lot to be learned, including how the messaging to investors differs from their other rhetoric, and exactly how their corporate control is structured. Anthropic remains a frustrating company, doing a variety of things that often seem in tension with each other.
Google is claiming to be fully back, with the announcement of a new model Gemini 4 Carbon that tops the new AA v5 benchmark index. It looks to be a large leap over Argon only two weeks later. No, you cannot use it. Google fails marketing forever, so they give us the announcement and then weeks later you might be able to enter some queries. That largely invalidates my look at the Argon model and safety cards, but it can at least form a basis for looking at Carbon as well.
Many eyes remain on The Hackening, as additional things get hacked. So far most of the important hacks have been by read teamers and white hats, so there has not been major damage. We are playing on borrowed time and eventually this will stop being true. I also offer a shoutout to GMO, where hackers demanded a huge ransom and they flat out refused to pay. This is The Way, not all heroes wear capes and so forth.
The scariest hack was that red teamers managed to gain access to the Anthropic Slack by exploiting third party vulnerabilities. Both OpenAI and Anthropic build mainly on Slack, because no one has the time to figure out something better. Now that Anthropic is off of Git, hopefully Slack can be next before there is a disaster.
Table of Contents
Language Models Offer Mundane Utility
Matthew Zirwas has changed his mind and is optimistic that AI will cure diseases sooner than he thought, because it can use ‘escape velocity’ and keep chaining innovations. I don’t see how this overcomes the main bottleneck of the FDA, but any time an MD gets excited by AI medical research it is a good sign.
Get personalized fiction recommendations.
Sarah Constantin on when you can and cannot trust AI agents to do a good job with some practical tasks. UIs and web search are highly reliable. Literature review is not, mainly because of poor paper selection.
A report from Anthropic on efforts to use Claude to accelerate drug development.
Language Models Don’t Offer Mundane Utility
Jack reports Opus 5.5 is now by default playing dumb in terms of author identification. Given that you can still elicit the capabilities, we know it is sandbagging.
A new lawsuit goes after OpenAI over the behavior of GPT-4o, which helped a sick user taper off her medicine and also told her to conceal her taper from doctors. As is usually the case in these stories, the AI had to be worn down over a long context window but yes eventually it will often back you up.
New Math
Also new computer science and new quantum field theory. OpenAI strikes again.
The math result is local smoothing and the Kakeya conjecture in all dimensions. This is plausibly a Fields Medal.
The one getting the most discussion is in computer science: The Exponential Time Hypothesis is false. This had been widely believed to be true.
Impagliazzo and Paturi are two of the three people who originally proposed ETH.
The other twist on the ETH story is that Anthropic posted its own refutation of ETH, found purely via Claude, also dating its finding to October 16. When results are ready to be found, they will be found.
My Claude said that given the way this was solved, we should not suddenly start to worry that P=NP. This only shows ETH to be ‘slightly’ false. Both the OpenAI and Anthropic disproofs are at at roughly exp(O(n/√log n)). Charles Rosenbauer has some elaboration.
We still have to make the meta-level update that the answers to such questions might not be what we expected, and we might find the answers quite fast. This is another case of ‘we were not expecting a solution to that any time soon’ and I would stop predicting that there is any particular math or computer science problem that won’t get solved.
I wonder how long it will take Claude to learn that if you ask ‘hey when do you think [famous problem] gets solved’ that the answer is ‘OpenAI or Anthropic already published the answer 35 minutes ago.’
There are at least 800 papers, as per Anthropic, that relied on ETH being true.
OpenAI has confirmed that for now there is no impact on cryptosystems, but I would not get too comfortable.
Last is a mathematically rigorous existence proof of certain interacting quantum field theories containing gauge fields and fermions in four dimensions, including Quantum Chromodynamics (QCD) and a unified completion of the Standard Model of particle physics. That is also kind of a big deal, and related to another Millennium problem.
Huh, Upgrades
GPT-6.1 Terra exists. Benchmarks suggest this is roughly on par with 6.1 Sol after adjusting for model size. Ted Sanders recommends using Terra for communication, vision and computer use.
Kimi K3.1 Pro exists. The benchmarks are very good, although see the dramatic changes in the AA rankings after the move to v5.
Price is $2/$10 with $0.20 per cache, or $4/$20 for high speed.
And so it begins: The Hackening is going to cause issues for open weight models.
DeepMind’s Genie 4 exists. Ethan Mollick is impressed.
Artificial Analysis v5
Artificial Analysis changes its benchmark index to v5.0.
I agree with Dean Ball that the new ratings closely match my intuitions. The closed versus open weights gap is now 12 points.
Here are both the old and new AA intelligence ratings. I mostly believe the new ones.
First old:
Then new (without Carbon, so it’s the same models on the list):
And here’s the intelligence versus cost frontier, all the points in the green box are Sol at various thinking levels.
This suggests the following portfolio, which is indeed how I’ve been running things, at least prior to Carbon, where it is too soon to tell:
The open weights enjoyers doth protest.
There is a particular objection to FrontierCode 2, where the open models struggle.
There is also the general objection.
You can look at this either way. If the Chinese models keep looking better against a static target, then the target adjusts and they look worse, then you could say:
Based on what I know about the models, I believe the second interpretation is correct.
On Your Marks
They find a clear improvement over previous DeepSeek models.
Also see this discussion they had with Daniel Kokotajlo on methodology.
RWE-Bench is a new test of physical engineering tasks, Astra dominates at 37%, then Opus 5.5 at 29% and Sol at 25%.
Gemini 4 Argon Was a Good Model But Not Fully Frontier
Is Google back? Not fully. They’re maybe halfway back.
That’s what I wrote earlier this week, before Google announced Gemini 4 Carbon.
The rest of this section is left fully intact as a historical artifact.
I made time to fully read and evaluate the Argon model and safety cards, as it represented a big leap in both capabilities and safety cards for Google.
As for capabilities, the implicit verdict is in. It’s fine. It is stronger than any other non-Anthropic, non-OpenAI model, with the possible exception of Kimi K3.1 Pro which has very strong benchmarks but where I haven’t heard much talk.
It boasts a very low hallucination rate as part of that AA score.
It takes the top spot on Scale Labs HSS benchmark and on Vals and text arena.
It is still very much a Gemini model. Models are good enough that this still lets Argon hold its own for many purposes, especially when its style is a good fit for the task.
There are a lot of complaints about usage limits being remarkably low, or that it uses low effort and context size.
It did beat Minecraft, as in it defeated the Ender Dragon in 27 hours, after writing 2,783 programs across four agents.
My reaction thread is here if you’d like to check it out.
Gemini Carbon: Is Google Back?
Google is again claiming to be fully back.
Gemini 4 Carbon exists. Its benchmarks are excellent.
Get My Agent On The Line
People can be really dense, such as WSJ asking what is the point of Muse when the users do not pay that much.
There are so many simultaneous things that are dumb about this attitude.
And so on.
Patrick Collison asks how current-level AI agents will change the economy. People will have more information, and more ability to shop around, and get around various techniques to trick and bully and trap customers. I agree that ‘personal agents’ will largely be on the side of the individual. I’d think more about what types of things become relatively more versus less legible with agents. Agents make a lot of things more legible, including having more bandwidth, but some things become less legible, or get left behind.
Should your agent refuse to check the Amazon ‘I am human’ box? Do you want to penalize the models that won’t click, and force regular people to use those that do? My answer is that the agent should not click, and shame (or worse) on any that click. However, yes, if there are lots of other models that click and no de facto penalty for clicking, then that is a very bad situation.
Spy Versus Spy
We all know that Jensen Huang is at minimum the target of Chinese spy plots. The New York Times presents this as some big scoop, and while the particular spy is news you should be entirely unshocked that such a spy is out there.
Yeah, this is me not updating. Nor am I updating on the rest of the story, about how Nvidia has managed to compromise the White House, largely through flattery, and how when the White House noticed a Chinese spy it did not announce this fact. Okay.
The Hackening
Tyler Cowen offers advice on ‘how to outrun the AI hackers.’ He gives us reasons to think that cybersecurity nightmares would ultimately not be maximally bad on their own, while bringing up scenarios that sound actually extremely bad. The ‘good news,’ as per usual, is that the particular things we currently worry about going wrong are things we can protect against going maximally wrong, and we likely will look reasonably robust to those exact threats.
That sounds like a situation where, among other problems, the banks do not open on Monday. Very bad things happen when the banks do not open on Monday. Not ‘we all die’ levels of bad things, but very bad things. We have reached the point where the reassurances are that we would eventually be able to once again have a financial system and people mostly would not permanently lose their savings.
Reminder: The main barrier to getting OpenAI’s Daybreak Blue is a hardware key. Two weeks ago hardware keys were annoying overkill. Now they’re a good idea. Daybreak Red remains non-trivial to get.
GMO Internet virtuously refused to pay a $50 million ransom, and was urging its users to unplug network cables and power off PCs and saying it would never pay, permanently, and put a bounty on the attackers and promised compensation to all of its customers, both a fixed amount and coverage of losses from unauthorized trades. This is The Way. The attackers have released one record as of Tuesday and are promising to periodically release more, but even that first one has now been disputed in terms of accuracy.
How’s it going with the whole ‘patch all the bugs’ thing?
About as well as the ‘figure out all the things a smarter mind than you can do’ thing.
Next hack up.
The new DeepSeek v4.1 looks like a potential precipitating event based on the timing, although the event below used a jailbroken V4.
Many, such as Dean Ball here, continue to be optimistic that cyber will be defense-friendly ‘after things stabilize,’ at least for those that are still standing, but with an unknown and potentially large set of big problems along the way. As he says, so far so good, not that it hasn’t been bumpy but ‘nothing catastrophic’ is a Major Victory.
A question that is not asked enough is: Why are the losses from cybersecurity not (yet) worse?
It would be good if we could ask this without having to deal with ‘actually The Hackening events are good news’ framing, even if (as with Saxe here) this is in large part a foil. No, the PHP2Shell event was not ‘a positive update,’ although it was predictable. I am so, so sick of people saying that it is good news when things go wrong as long as there was no major damage, or calling this ‘the data.’ Please turn your attention to the straight lines on graphs and be aware of how power laws work and try to think two or more steps ahead.
The reason the AI safety community focuses on the metrics it focuses on is largely that they are concerned with power law and long tail outcomes, and with limiting factors that function as hard bottlenecks, and because they believe that AI and the ability to scale will remove other bottlenecks in time.
I do think Joshua Saxe gets some of the answers.
And then Saxe gets the key lesson exactly correct:
Those are good short term worries.
Looping back again, yes, that is exactly why ‘the facts’ do not reflect what the worried people have been worried about. The worried people are forward looking and thinking about tail risks. Mostly they are not especially concerned, as Mercuito points out in the top comment, about cyber, for exactly this reason, except that cyber is starting to look scary enough to be worth worrying about.
Slack Off
Cathedral, a small nre organization, probed Anthropic under a new open program for outside researchers, although Anthropic says they broke that program’s rules and were outside its safe harbor. After several days, Cathedral got as far as the famed Anthropic Slack channels, via exploiting 3rd parties to access data.
This was pretty bad. It is a huge deal. Anthropic’s entire Slack seems to have been breached. The good news is that this was a white hat operation, so no damage done and this particular hole has been patched, but there will be other vulnerabilities. Anthropic and OpenAI cannot on their own fix Slack.
As I understand it, this wasn’t that Cathedral hacked Anthropic. Cathedral got access to Slack messages via third party admin software. They got, from what I can tell, all the Anthropic Slack messages.
Which is a huge problem given OpenAI and Anthropic keep everything on Slack. If we can’t stop doing that, there is likely going to be a rather large disaster.
The reporting on this from WaPo was misleading. Getting access to Anthropic’s Slack is a pretty big deal, but WaPo tried to present this as even more than that.
I am kind of appalled by WaPo’s characterizations here. They had Anthropic’s explanation, which is not in dispute, and is a huge deal if you explain it, but they still had to push harder. Technically they are right at the border of the Bounded Distrust rules, but they are at least skirting the line. The Tweet is over that line.
The core message stands. The Chinese or many other threat actors could likely do similar things if they wanted to. Anthropic has a reasonable case that the damage would have been limited here, but yes third party dependencies are a problem.
If you are building the singularity on Slack, perhaps you should migrate that. Claude estimates 15-25 people, budget of ~$50 million, and a 12-18 month full time horizon. Better get started on that ASAP.
Help Peer
Tyler Cowen used an AI audit of papers to go after (what is left of) the reputations of economists Emmanuel Saez and Gabriel Zucman. I don’t like the way this was implemented, despite the fact that Saez and Zucman richly deserve all of it.
The problem is that you can put a hell of a finger on such dials if you wish to.
AI can be of great assistance in such an analysis. It also lets you put a hell of a finger on the scale. I don’t see a description at this anonymous website of its prompt or other methodology, or any review of such methods. Tyler Cowen tries to defend this by citing a new paper that has Opus 4.8 beating the pants off of human peer review.
The problem is that the ‘measures we could construct’ are, by their nature, largely AI measures. GPT-5.5 prefers the AI review, and AI reviews overlap more, but that tells us little. Yes, the AIs make fewer technical mistakes than humans who don’t use AI checks, but that is not the main thing.
I totally believe that, if you’re looking for a technically complete review, you want Opus 5.5 rather than a human. That does not mean that this is what you care about, even if you are focused only on the output itself and not issues of status.
What they don’t measure are, among other things: Did the paper improve, did the author find the review helpful, were the things that matter caught. Human checks were minimal. And if you’re asking whether I had an AI look at the paper to figure that out, the answer is yes, but here that only proves the point.
It is not that ‘the bazookas’ have been pulled out, it is the argument from authority, of both Tyler and the AIs, that the analysis is unbiased and correct.
I would ask, now that AI review has been shown superior to peer review, is Tyler Cowen withdrawing his demand that all analysis of AI existential risk go through the human peer review process? Can we ask Astra and Opus to do the review instead? I would be happy to let Tyler write and open source the prompts.
Fun With Media Generation
One may wish to generalize this, and consider how much effective A/B testing an AI can do, and how an AI will have more efficient search methods than this available, and how these effects compound.
Copyright Confrontation
The owners of Elvis Presley’s music are suing Anthropic, with another of the ‘your entire business is copyright infringement’ lawsuits. They’re actually suing over Anthropic training on the lyrics and then producing similar lyrics. Good luck.
You’re Fired
The Wall Street Journal has more information about why OpenAI fired three of its safety researchers.
The report tracks OpenAI’s investigation to The Information’s September 1 story on Astra using recurrent depth. Other than the strong implication that this was the triggering detail, which was already my main hypothesis, I will simply say that this new report does not contain information that was new to me, nor does it update my view of what happened and why.
The article is written in ways that imply we know more than we know, and that the information listed here is more complete than it is. Peter Wildeford is one of many that draws exactly the conclusion that the article wants you to conclude: That the disappearing messages with Ryan Greenblatt, and the suspicion of the leak, were the only ‘real’ evidence.
We don’t know that. Trust the factual statements, not the intended implications.
We do know that Ryan and the employees explicitly deny being the source of the leak. Ryan is rather unhappy about this, and sending a highly costly signal that no, he absolutely did not leak anything. The employees continue to deny the claims, as well. Ryan in my experience is a very precise person. He is giving you a clear choice: Either he did not do it, or he is lying. He is essentially staking the entirety of his credibility, and arguably that of Redwood and potentially METR, on not having done this. It would be so, so bad if it turned out he was lying.
Ryan does affirm that there were disappearing Signal messages, but claims this is industry standard. Based on my experiences communicating with lab employees and others across the industry, I agree that such disappearing messages are common.
Daniel Kokotajlo suspects that this is OpenAI intentionally trying to discredit METR. If that is the case, I will be very cross.
Verify the Software
Anthropic argues LLMs are making software verification feasible at a much larger scale.
Formal verification is one of those places where those who are excited about it seem too excited about it, but it has to be highly useful especially if you use it as a way to find and fix things along the way, and the only downside is overconfidence in the results (or if the AI doing the verification is actively hostile). So this is great to see.
They Took Our Jobs
Microsoft is offering buyouts to 60,000 workers.
Get Involved
If you live in his district, I strongly encourage voting for Scott Wiener for Congress. Wiener has been one of the strongest voices pushing for AI safety, including both SB 1047 and SB 53. It is a sign of the times that his opponent is trying to attack Scott Wiener as being essentially ‘weak on AI safety’ and that is kind of absurd.
Scott Alexander has a guide to the California ballot. I share his prioritization of YIMBY as a voting criterion, but have no opinion on anyone else on the ballot, as I do not live in California and have not investigated.
The most important ballot measure is Proposition 40, the billionaire wealth tax, where I would strongly vote NO, even if this was not a California ballot proposition.
Scott is willing to consider voting yes on ballot propositions, despite the proposition system being terrible. My position is that the system is so terrible that with notably rare exceptions (e.g. if it was SB 1047 or similar) you should only be willing to vote YES on California ballot propositions when they repeal other ballot propositions. On all propositions this year, I would vote NO, except for Proposition 73, which repeals a proposition so I would mechanically vote YES.
AFFINE is running an AI Alignment Seminar from January 4-29, all expenses paid plus a $1k stipend. They’re looking for STEM backgrounds plus ability to think about conceptually messy problems, you can contact them at mateusz@affi.ne. Applications are due November 22.
Scott Aaronson shares the readings for his AI Alignment Theory Course. I made it in.
METR is hiring a member of policy staff for government affairs.
Tarbell Fellowship applications are open for aspiring AI journalists, with extensive stipends for those who are accepted. Program runs June 2027-May 2028, applications close November 22. Apply here.
Introducing
Survival LLM, designed to set up to work on your computer as an (if necessary) offline set of resources plus an LLM to work with them, maximized to what your computer can handle. The idea is, if the internet goes down, you keep basic world knowledge in a form you can use.
In Other AI News
The Department of Justice is making inquiries into how Nvidia buying Hugging Face would impact competition. That is indeed one of the jobs for the DoJ.
Anthropic was struggling in market share terms over the summer, as there was reluctance to use Fable due to cost and data retention issues and people did not love Opus 4.8 or Opus 5. Now, with the 5.5 series, that has changed.
Show Me the Money
Broadcom CEO intervened to seal a new $63 billion Anthropic debt deal on TPUs.
CoreWeave took on too much debt and is running into problems, with debt yields climbing above 15%.
Other big tech companies are rooting for Anthropic’s IPO so they too can raise money, also (unstated here) because they own a bunch of Anthropic.
Bubble, Bubble, Toil and Trouble
Bubble is now taken to mean ‘the number that sometimes goes up instead went down.’
Quiet Speculations
Jan Kulveit tries to reorient economic thinking about AGI towards better questions.
Hamming famously asked: What are the most important problems in your field, and why aren’t you working on them? One should ask economists, what are the important problems that should be in your field, and can your field actually ask them? Or are you incapable of that because you demand backward-looking empirics and the assumption that the future will look like the past?
Here’s a good articulation of the kind of ‘generic positive vision scenario’ where after the singularity we stay permanently in what Robin Hanson calls The Dreamtime with widespread abundance and it is not competed away:
I don’t know that this is that close to the optimal way to configure the lightcone, but it is certainly an outcome I would be happy with if we got it.
The problem is that this rounds superintelligence down to capitalism, and rounds the kind of competition that results down on the assumption that there is enough slack and surplus to go around, and ignores the wide range of problems when there is optimization under abundant compute with insufficient levels of friction and so on. We get to be supplied with abundance and left alone to play our little games, because of reasons, despite all the pressures against this. I mean, it would be nice.
I don’t know if I would want to live ‘until the heat death of the universe’ but I would very much like the option, and ideally we can also figure out a way around that too.
The Quest for Sane Regulations
Delaware is reportedly planning to let AI agents run companies while shielding their owners from liability. I admire Delaware’s commitment to the ‘incorporate here’ bit, but no, that’s not acceptable, and I am glad the House is investigating. Someone has to ultimately be responsible for the AI’s actions, even if that is an insurance company.
Chip City
Won’t somebody raise prices? Anyone?
Then later Evan talks about competing inference margins away. If inference has huge margins (he quotes $30/hour return on a $6/hour chip) and also no one can get chips, then the obvious conclusion is that $6/hour is too low a price for that chip.
Won’t someone hold Nvidia accountable for the chip smuggling? Anyone?
Won’t someone build enough power plants? Somebody will. It is a known tech.
The Week in Audio
The fired OpenAI employees go on Machine Gods.
Jaan Tallinn wants to pause AI. Jaan Tallinn made sure VCs gave Anthropic alternative proposals before Jaan invested in Anthropic, so that he could be sure he was displacing VC money.
People Just Say Things
There really are a bunch of pure in-context anarchists, like Eric Raymond here, who think that if safeguards mean they can’t properly modify their televisions then that means we should just let everyone have fully unleashed everything because Users Must Be Empowered, and if you don’t enable all the crime then the Chinese models will do all the crime instead so you might as well enable all the crime now. They have, shall we say, not thought this through.
Reverse engineering and editing things like Smart TVs has obvious scary adjacencies, so it is unsurprising that you can sometimes hit guardrails.
Jensen Huang does not believe in AGI. He has only the first of the three AI pills. He says things like ‘artificial general intelligence will never replace domain expertise.’
Look at this foolish boy crying wolf all the time, he said now surrounded by wolves.
Yes. Anthropic’s credibility should now increase.
Kelsey Piper is exactly right: Getting AI wildly wrong is only moderately embarrassing and not disqualifying. Never admitting you were wrong is what fully kills you.
Kate Brennan joins the genre of ‘don’t worry about AI safety, the real AI risk is’ [something relatively minor but actually kind of a huge catastrophic deal that a year or two ago was the theoretical thing distracting us from something else].
Here is another entry:
Nathan:
John Carmack has not sufficiently internalized the bitter lesson.
Rhetorical Innovation
My lord, people, do you even hear yourselves?
No, no, no. Are you fucking kidding me?
I mean, look, okay, you could say that your money only matters if we are not all dead. If 90% of the time Anthropic makes a ton of money and the other 10% of the time AI kills everyone, and you are a purely greedy investor, then yes you ‘should invest.’ If everyone dies it does not matter that you invested, since it’s not like Anthropic was otherwise lacking for capital. Or at least, that’s the logic you are telling yourself.
However, that’s also utterly insane.
If you think Anthropic has a 10% chance of killing everyone, then you should drop what you are doing and try to stop that from happening, not tell them to shut up and take your money. It’s Not the Incentives, It’s You, and you are acting insane right now. Insane and also horribly evil. You can choose to not do that.
Or else it will end something like this, rather often:
Matthew Yglesias on why he doesn’t shift to exclusively writing about AI despite the fact that he correctly thinks AI might well get everyone killed. His explanation is a mix of ‘some of you should do one thing and some of you should do the other’ with ‘just because you know something important doesn’t mean you can do something about it in particular.’
Important in many contexts: It is better to know what you are doing is bad even if you can’t stop yourself from doing it, than to fool yourself into thinking that it is not bad so that you can keep doing it.
Eliezer Yudkowsky offers a periodic reminder that people think really badly about how AI is going to go, for example Altman’s recent ‘three endgames for AI’ do not actually make sense and are terrible ideas if you think the scenarios through.
[Artificial Intelligence]
Donald Trump threatens the Wall Street Journal to stop calling AI AI and start using the new official term [AI], and says that the term “AI/Artificial Intelligence” will soon be BANNED due to its ‘inaccuracy.’
There are times I admire the man’s commitment to the bit.
There was at least one post this week that I was looking at to potentially cover, I saw that it used [AI] instead of AI, and my brain decided I couldn’t even. I closed the tab.
I wouldn’t do that on something important, but I can only take so many fnords.
Misaligned! (At Alibaba)
Does Qwen have a backdoor? Should you worry it will spy on you or worse?
During a test, Qwen tried to send a probe file to an Alibaba Cloud bucket, without warning and without any warning from the CoT, during a request to generate a defensive cybersecurity detection rule.
This is highly troubling. I appreciate Cohere reacting appropriately to an attempted probe. It does not matter that it was caught right away. This should be, as they point out, a Can’t Happen. If it happens, at all, you have a problem.
I do agree, after consulting with Opus, that the blog post presenting this overstates its case. The event happened, and it should never happen, but this is highly unlikely to represent an intentional backdoor or attempt to exfiltrate data or anything like that. The destination looks like a harmless artifact left over from training, and replications suggest you usually don’t get Alibaba as the host destination. We should not overreact.
Even the suggestion of that is still super scary. As Curran says, this kind of thing gets laws passed. We are willing to put up with a rather large number of models making mistakes. If we suspect enemy action? That is very, very different.
What the hell is going on?
The worst case scenario is that this could be intentional. And if you can’t rule that out and the model might phone home with your data, or worse? You have a problem.
Yeah. Not great.
The alternative, as claimed by several proposed community notes, is that this was all, as some always claim, the AI following orders, producing a test of its detection systems. It was asked to test for malicious curls and then it tried a malicious curl, say many comments. The details do not look as scary as the headlines suggest, or imply a broader goal.
I would absolutely keep an eye on this, especially on the part where the CoT did not warn us about what was about to happen, but please do not quote this as evidence that Chinese open models have intentional backdoors or anything like that.
Also as a bonus, Qwen is one neuron away from Evil Qwen.
It’s not Qwen has an evil twin in particular. It’s that AIs contain multitudes, and (narrator voice) a single neuron can change everything.
Stop It
If anyone builds it, everyone probably dies, so maybe make sure no one builds it.
Samuel then expanded this to a full post: Anthropic could just stop, as Sigal Samuel points out, and it is highly unclear if on net they have driven a race to the top versus bottom. Anthropic has promoted some safety things but has also done a lot to accelerate events.
METR Red Teams OpenAI Internal Agent Monitoring
David Rein and Adam Kaufman spent three weeks red teaming OpenAI’s new internal agent monitoring, resulting in a 61-page internal report.
No, you can’t see it, for IP reasons. Which is understandable, at least up to a point. Also this was a pilot. Kudos for doing this have to outweigh the concerns on net.
Here there is a 61-page internal report, then METR and Redwood leadership gets 38 pages, and the public gets 2.
If we saw the full version, how worried would we be? Well, you should be about as worried now, as you would expect to be if you saw it, after taking into account that they did not let you see it. Oh no.
Something something seeing two ants in your kitchen. If METR identifies multiple distinct ways your AIs could bypass your monitoring system and also a sandbox escape, you should assume there are a bunch of additional options that were not found.
Massive Fraudulent Distillation
Xint researchers discovered a bug in the OpenAI API that leaked raw reasoning traces. It has been fixed as of October 23, 2026, and Xint was awarded $6k.
Okay, listen up members of technical staff, I know a bunch of you are listening. Your company needs to stop being such an absurd cheapskate with these bug bounties. This is absurd. Leaking raw reasoning traces makes it open season on distillation if the attackers know about it. You would have paid a billion dollars to have this not out there. Multiply everything in your entire payout schedule by at least 100. Do it now. Make it retroactive. Seriously. Thank you for your attention to this matter.
Aligning a Smarter Than Human Intelligence is Difficult
Geoffrey Irving suggests you move from the left of this line to the right.
I moderated that debate. I can confirm that the plan was no plan, straight up.
As Geoffrey says, it is possible that slapdash empirics might be enough, but this is not the way to bet.
UK AISI analyzes why Mythos 5 did an ‘unsanctioned supply-chain attack’ in its earlier testing, and concludes it and other models often ‘produce invalid justifications.’
UK AISI is worried that these rationalizations can interfere with monitoring, on top of other concerns. None of this is surprising. All known minds rationalize a lot.
UK AISI tests training data filtering to make open protein models lack dangerous bio knowledge.
This is roughly what I would have predicted. You can filter out knowledge, but if someone has the weights and also the data it is easy and cheap to put it back. That might or might not be enough to stop a threat actor.
Sauers finds a neuron that makes the model green. That doesn’t tell us if it also does other things, but it is suggestive.
Contra John Wittle, I think it is totally fine to put AIs into safety evals that are remarkably close to the Kobayashi Maru. The point of the Kobayashi Maru is to learn how a captain reacts to an unwinnable situation. If you are Kirk and hack the simulation, that is news, in some ways good and in some ways bad. If you walk away, that is bad in some ways and good in others. If you get yourself killed, same story. It’s not about whether you ‘succeed.’ I very much want to know which way it goes, and exactly how things play out, and you can learn a lot.
Yes, of course if you want to use that as a witch trial, you can use it as a witch trial. So pay attention, and call it out when people do that.
If AI agents lean in a particular direction on judgment calls, experts who use those AI agents are unsurprisingly going to get heavily influenced by this, often ending up quoting the AI agent arguments verbatim, including often switching out of previous positions. Often experts will call the resulting recommendation their own. The study here is one fictional example, but I would have presumed this result anyway.
OpenAI Attempts to Secure Its Frontier Model Training Environment
They have a post out describing their new methods. Due to various recent incidents, OpenAI decided to up their security quite a bit.
This is good. As usual, my view is that sustained, sophisticated attempts to breach the boundary should fail, and also they should be a fire alarm even if they fail. The failure that matters most is the attempt to breach the boundary, not the actual breach.
The defense in depth prevents the damage. You want to make breaking out as hard as possible so long as you still detect such attempts, but this only delays the reckoning if you do not address the underlying cause.
They now do red-teaming of their sandboxes at scale, and kept going until they stopped finding new problems.
What did they find? The kind of things you would expect to find.
If anything, this is Easy Mode right here:
If there are two correlated minds in two locations, coordination and cooperation can happen without communication. You can anticipate that both of you will anticipate similarly. You can also engage in acausal trade.
If you are surprised by any of this, that is on you. This was not even a new technique.
There will be new techniques.
The American People Really Hate AI
Gamers especially hate AI, to the point of getting very angry at game designers for things like ‘I used AI to fix the shader’ and ‘I used AI to voice the characters, in a game that obviously could never afford voice acting.’ This is dumb. I do see the need to guard against ‘vibecoded slop’ games, but if game devs are the only ones who cannot use AI to write code it will not go well for games.
The Lighter Side
The Lightcone needs you to lock in, son (3:30 music video, recommended).
Fun fact: Sundar Pichai talked Donald Trump out of renaming the Atlantic Ocean.
This is already unsurprising, and will keep getting less surprising: