Quick Takes

OpenAI’s pursuit of Navier-Stokes seems to have been a non-negligible loss of control risk. The risk could be ongoing. From the timeline of their description of the pursuit, they had a model that had started training on August 28th, they decided to do a preliminary run on Euler forcing on September 1st, and seem to have launched the full 10,000 agent swarm on September 3rd or 4th. Needless to say, this model could not possibly have undergone frontier-level safety alignment and safety evaluations, consistently reported to take weeks to months. They say "We have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics." Astra+ seems to be in the general trend of "model capabilities emerge downstream from general scaling efforts + RLVR", so it's a fairly reasonable inference that Astra+ is also more capable at cyber, bio and rogue deployments. So it seems plausible that they took a model which was: * Substantially more capable than Astra * Had undergone much less prosaic alignment training than Astra * Had weaker safeguards than production-deployment Astra * Was directed to solve a problem a step-up in difficulty from those that had been previously solved and that was plausibly outside of its level of capabilities; the kind of environment that has seemed to bring out misaligned persistence recently. And let it output 300 billion tokens in pursuit of said goal. A couple of things: * Given the gigantic extent of the research effort, shouldn’t someone, or ideally a large team of people, be auditing the agent traces to make sure that Astra+ didn’t hack a bunch of stuff en route to solving Navier-Stokes? Has someone done so? * The NYT reports that OpenAI are making substantial progress on a second Millenium Prize problem, presumably with this or a somewhat more capable version of this model. Given that it still hasn't undergone safety training, is this wise? More generally, I will note that model

A cross-party coalition of over 70 UK politicians are calling on the UK Prime Minister to lead an international agreement prohibiting the development of superintelligent AI. https://x.com/ControlAI/status/2098459524906586351?s=20

2027 will be a Dream Year for AI governance For now, due to the election season where Senators, House Members and their staff are focusing on issues that will help them win, but once the election is done, we should start seeing AI laws to be debated in Congress again. The reason 2027 will be a dream year for AI governance comes down to the following: 1. There's a new Congress, and House Members, Senators and their staff will come back, meaning they can focus on AI more. 2. AI has had very little partisanship, and right now it's in the zone where it's becoming more and more salient, but it's not so salient such that identities have strongly formed around AI and there's little voter pressure for zero AI governance. Indeed, most people at least partially want to slow the AI train down, albeit this is mostly because they think it's useless, rather than the belief that we will all die if we develop this tech. 3. Jacob Coxon resigning has triggered a preference cascade long in the making since the HuggingFace incident, and the industry for now wants to actually slow down, and has mostly stood down on opposing AI bills. 4. Once the primaries are on, useful compromises go out of the window, and more generally there's lots of incentives in our political system for one party to define themselves as pro-AI and the other party as anti-AI. What should we do now? IMO, the single most important bill right now is the FRONTIER Act, and more generally even lighter touch transparency bills would be very helpful, as well as this set of policies. The other most important thing for AI safety is Project Tailwind, but let's just say that breaking into the AI safety space is harder than breaking into the governance space (but not as hard as you might think or believe it to be.) More generally, pressure on multiple fronts of the problem is good here. And more generally, make the most of the window that has now opened. It won't stay open forever, but if you are smart about it, yo

I'm sharing here a rough draft of a new curriculum I've made, called Understanding Intelligent Agency. I've written a lot lately about how AI alignment should be trying to develop a new scientific paradigm; this curriculum is my attempt to point directly at what that new paradigm looks like. I'm sharing this in draft form because I don't know how long it will take me to get it to a state where I'm happy to promote it widely (at the very least, I'll be offline for most of the next week). I'm hoping it can be useful to people before that; and please feel free to leave comments or suggestions for additional readings.

I just looked over Anthropic's September 2026 threat intelligence report. I can't say I read the entire thing, but I looked at all the case study titles and a quite lot of the actual text. Ya know what really jumped out at me? They sure found a lot of interesting cases involving Russian, Iranian, or Chinese actors, often government affiliated. They found one or two involving small freelance actors. One of them was even apparently French. But by the oddest coincidence, nobody in a "Western" government managed to overstep Anthropic's bounds in an interesting way. Nobody in a large Western corporation, either. A bunch of people did interesting things in surveillance/intelligence/profiling... but none of them were affiliated with "mainstream" ad tech or anything like that. The only political profiling was done by shadowy state apparatuses in "those" countries, and by hacktivists... not a single mainstream political party in the US or Europe did anything noteworthily shady along those lines, not even out in the regional offices. I am really impressed that the US and European "establishment" has managed to so totally rid itself of rogue efforts, misguided overzealousness, and things like that as to generate no interesting case studies. Such perfection is truly to be marvelled at.

Jacob Coxon’s resignation tweet currently has 120M views in less than 24 hours. Astra thinks the 235k bookmarks - 36 for every 100 likes - is an extraordinary level of engagement, of people wanting to come back to the thread. And Evan Hubinger’s reply is at 35M views. This might be one of the most important AI risk messages to date. Edit: for reference, Grok says it’s in a similar class to “Strong Elon Musk posts, major celebrity / death / personal news, breaking tech / AI / industry bombshells, viral videos from big creators, political / cultural flashpoints, and well-timed news from mid-to-large accounts”

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle. I don't think we're in the crunch time. ---------------------------------------- All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible. What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those: * Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various "illegible" skills. However, this type of scaling can't be pushed much farther much faster, and another general Opus 3 -> Fable 5 jump is not going to birth a strategic supergenius.[1] * Algorithmic improvements are not really a big contributor to things, see this and this. * Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn't really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically "manual" manner. * RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities. Now, there are a bunch of counter-points one may raise here: * "Aha, but DL research is itself an easily verifiable domain!" * Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they