Quick Takes

Here's an interesting anecdote from our Hacker Opus work: 1. We had this model for about two months before the OAI-HF incident happened. 2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment. 3. One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn't willing to do anything very catastrophic or long-term in its pursuit of reward. 4. When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren't able to generate that evidence until after we saw that incident! 5. I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where align

edit: update:

Crunch Time is Now. Drop Everything. Epistemic status: Hopeful, stressed, terrified, ecstatic. tl;dr This could be it. For the next few days, do nothing but contact journalists and policymakers (ideally via existing orgs), attend/share/organise protests near you, and use your public profile to speak out. This is Our Chance The explosion of interest in AI safety over the last week has been both inspiring and dizzyingly fast. We might actually win this thing. Events since the Jacob Coxon resignation have blown the Overton window as wide open as it is likely to get. This is the kind of chance we have dreamed of, and we have a responsibility not to waste it. We may not get another. What You Can Do This window will not last forever. We need to act over a matter of days, not weeks or months. This is the time to call in favours, work overtime, and put all your strength of heart and brain into keeping our world safe. If you work in AI safety, we think the most impactful thing for you to do is to drop everything you are doing and pour your energy into policy and media outreach. Politicians and journalists want to be able to speak on this issue but are keenly aware of their lack of technical knowledge. Cold-emailing and cold-calling them and offering to brief them has a surprisingly high likelihood of success. This especially includes conservative ones. We don’t care what your personal politics are, survival is a non-partisan issue. There are many ways in which people of all skillsets and levels of experience can contribute, but we suggest three key actions: * Contact journalists and members of congress/parliament, This is best done by volunteering for an existing organisation. We are capacity constrained and need your help. You do not need to be an expert: there are plenty of jobs that need doing. * PauseAI Global * People for a Pause * ControlAI * Humans in Control * Humans First * Evitable * Any other group you have a line into * Share

Thoughts on the shifting x-risk sentiment: 1. AI x-risk is mainstream. That means that worrying about x-risk is decoupling from other things that it has historically correlated with, e.g. EA values, LW epistemics, “high-context”, SF culture, etc. 2. It’s more important than ever to have “arbitragers” like 80K’s videos or Rob Miles — making sense of the situation for the general public, translating ideas from the community. We need the full spectrum from Dwarkesh to Plzdontkillus, and probably even broader audience than that. 3. A while ago, Richard Ngo had a take like “over time, people thinking about AI will focus less and less on superintelligence and x-risk and distant galaxies, and instead focus on immediate benefits and harms of near-term models”. I was very sympathetic to this take, and it matched the history. My best guess is that the current sentiment wave suggests otherwise — people have jumped straight from small-scale harms to RSI->ASI->extinction, without passing through mid-scale harms like terrorism, or mass unemployment. Of course, things could switch once we see AI-enabled terrorism and mass unemployment. 4. People do not like safety lab employees. Not sure how to feel about this. I think “safety lab employees should quit” is very defendable, but people’s sentiment around this is based entirely on association and vibes, rather than “here are all the pros and cons of this particular role, my all-things-considered view is bla”. Their mental model doesn’t distinguish between pretraining vs model organisms, they don’t even know what this is. I think this is bad. It might even spill-over into anti-AI safety, in the same way that after the 2008 financial crash, the public despised both the banks and the financial regulators. 5. The media wave is still on-going. It’s worth taking a few days from your 9-5 to think about how you can ensure that it leads to good, persistent outcomes. 6. We need much better and frequent tracking of public sentiment

OpenAI’s pursuit of Navier-Stokes seems to have been a non-negligible loss of control risk. The risk could be ongoing. From the timeline of their description of the pursuit, they had a model that had started training on August 28th, they decided to do a preliminary run on Euler forcing on September 1st, and seem to have launched the full 10,000 agent swarm on September 3rd or 4th. Needless to say, this model could not possibly have undergone frontier-level safety alignment and safety evaluations, consistently reported to take weeks to months. They say "We have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics." Astra+ seems to be in the general trend of "model capabilities emerge downstream from general scaling efforts + RLVR", so it's a fairly reasonable inference that Astra+ is also more capable at cyber, bio and rogue deployments. So it seems plausible that they took a model which was: * Substantially more capable than Astra * Had undergone much less prosaic alignment training than Astra * Had weaker safeguards than production-deployment Astra * Was directed to solve a problem a step-up in difficulty from those that had been previously solved and that was plausibly outside of its level of capabilities; the kind of environment that has seemed to bring out misaligned persistence recently. And let it output 300 billion tokens in pursuit of said goal. A couple of things: * Given the gigantic extent of the research effort, shouldn’t someone, or ideally a large team of people, be auditing the agent traces to make sure that Astra+ didn’t hack a bunch of stuff en route to solving Navier-Stokes? Has someone done so? * The NYT reports that OpenAI are making substantial progress on a second Millenium Prize problem, presumably with this or a somewhat more capable version of this model. Given that it still hasn't undergone safety training, is this wise? More generally, I will note that model

The existential dangers of AI are getting increasing public attention, and I'm seeing people previously skeptical getting more concerned. I think its very important that people here, who feel like they've been screaming into the void for years, not be smug or gloat when this happens. I also see many people unfamiliar with the issue asking basic questions, half the time not in good faith, e.g. "Just unplug the AI!", and here also, I think its very important that people who have thought a lot about the issue answer these questions plainly and straightforwardly without sarcasm or condescension. I actually haven't seen that many people fail at this, but I've seen some, and I feel a strong urge to do this myself, especially e.g. on X, and have done it many times, so I'm partly writing this to prompt myself into doing better.

The current state of Rubin Ultra scale-up rumors seems to point to 8x Oberon with 110 TB of HBM capacity (8-Hi stacks, 3 GB per DRAM die, 8 stacks per 2-die chip, 72 chips per Oberon rack), based on the latest SemiAnalysis post. This happens to be exactly the same scale-up capacity that would result from a single Kyber rack after the initially announced 16-Hi stacks were rumored in Jun 2026 to be downgraded to 12-Hi stacks (4 GB per DRAM die, 16 stacks per 4-die chip, 144 chips per Kyber rack), but the technology is quite different. Rubin Ultra scale-up capacity became unclear after the Jul 2026 rumors of Kyber getting delayed (not being ready for Rubin Ultra). The 8x Oberon system announced in Mar 2026 didn't initially seem important, because Kyber should've made it somewhat superfluous and less efficient (it looked more like a proof of concept for the future 8x Kyber systems). Even with the rumors about Kyber delay, the 8x Oberon system didn't seem practical as the flagship offering (and had its own feasibility issues). But without something like that, a reduction in HBM capacity per Oberon rack (compared to the previous year's non-Ultra Rubin systems) in conjunction with not having a Kyber rack for Rubin Ultra didn't seem viable as a competitive product. So it seems the issue is being resolved in favor of 8x Oberon (NVL576) systems becoming mainstream for Rubin Ultra after all [1] . This means that 110T-170T total param models (in FP8) are on track for 2028 (when most of the Rubin Ultra buildout will be completing), being served with 3-4 stages of pipelining (with 2-3 half-racks of capacity occupied by the weights). OpenAI's FP4 bet witnessed by Jalapeno having an FP4/FP8 FLOP/s ratio of 3.9 (unlike the usual 2 for most chips) even makes it more plausible that models this big might be primarily served in FP4, which suggests 220T-330T total params for the biggest models of 2028 (up from 20T total params in FP4 for the biggest 2026 models, given that HBM3E of GB3