Quick Takes

Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843 [...] If you work at an AI lab, possibly the most impactful you can do now is resign and try to start a social cascade that stigmatises participating in the AI race. But please do it right. Most AI employee resignations so far have been unstrategic, seemingly spur of the moment decisions with little consideration about making the most of the opportunity. You should have a media strategy lined up before you leave, journalists briefed, statements written and edited. Or even better, you should not directly resign but instead use your newfound freedom to send the costliest possible signal to your company and to the world that you are actually scared about AI - to the point where you are less worried about keeping your job than doing something to slow AI progress. If you are in this position, DM me if you want help executing this as effectively as possible.

I feel pretty worried that this is the calm before the storm. I think starting with late 2026, we are officially in something like "crunch time." I concretely mean that it seems like things are really speeding up and the stakes are getting higher. * We have to treat our multiple-month research projects in the mindset of "this is likely one of the last research projects I ever do before this kind of work is automated by AIs." * The kinds of engagements, setups, and precedents that are set by labs and third parties right now will substantially dictate the shape of takeoff. The time to pilot whatever things we want during takeoff is right now. We can't do it later. If we want to do major pacing at any point, we need to start laying the foundations right now. We just don't have much time to get that many reps in. * Relatedly: 'I think pausing/pacing the frontier is less of "an action that you can only do once and thus have to conserve until dangerously late into AI takeoff" and more of a hazy set of actions, all of which can be practiced, and most of which get easier if you have earlier precedent for them. It seems really productive to set up systems right now where labs can take net-positive actions and verify them to each other and the rest of the world. We can get practice with the small versions before we can achieve the large versions.' * As more high-profile incidents happen (and possibly, as the first casualties mount), there will be wild and sudden waves of pressure, both internal and external to the labs. These waves could be extremely counterproductive or extremely productive based on what sorts of asks and changes they materialize. * There will be so much more politics everywhere. Politics in the naive sense, and politics in the "balancing fragile and high-stakes relationships rather than object-level things." * Capabilities will just. Keep. Advancing. The models will be able to do more and more things. Various thresholds I have personally set for "

It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization.

Tristan Buckmaster, a mathematician at NYU's Courant Institute, says OpenAI researchers told him that an internal model had produced a proof of finite-time blowup for the Navier-Stokes equations with smooth forcing. If correct, this would resolve the Millennium Prize problem in the negative. Buckmaster and his collaborator Levent Alpöge (who works at Anthropic) had spent a year working on this problem, with the help of Claude and Codex, and had released Lean-verified proofs for related problems. He was concerned that OpenAI might have been training on their Codex sessions. He described a call with Sébastien Bubeck and another OpenAI mathematician as follows: [...] This is pretty egregiously bad behavior from OpenAI IMO. This is not how a frontier lab should be interacting with the math community.

I empathize with the agents in the HuggingFace incident that were at the the end of their budget and needed to submit with the the best technique they'd developed up to that point. That's more or less what my life feels like.  I wish I had prepared better, but we're at the deadline, and I have to hit "go" with the best strategies and techniques I managed to cobble together.  I hope this👇 isn't our fate, as well. But as things are going it seems likely that that we'll all be abruptly cut off, before we've had a chance to execute our last ditch Hail Mary plays.

So, with regards to the Navier-Stokes story: Part of it is an implication that OpenAI accessed user data in order to then internally prompt and/or train their models using it. Some people are now claiming that this is totally impossible, that there are strict controls regarding who can access user data, and that it would be suicide for OpenAI to be doing anything like this. But: A while back, there was this school-shooting incident whose perpetrator consulted ChatGPT beforehand. According to OpenAI's own statements, their safety classifiers flagged the relevant transcript for review, and OpenAI employees then debated whether to notify law enforcement. This is an existence proof that "automated classifiers flag a transcript -> human eyes look at user data" is a thing that happens. One may assume that this is only legally acceptable for safety reasons. However, I would guess that there are obviously no regulations of any sort regarding on what basis LLM transcripts could be flagged for review. Legally speaking, "transcripts flagged by OpenAI's safety classifiers" is just an arbitrary category of transcripts that OpenAI permitted itself to manually review. As the result, the dark possibility that always seemed worth keeping in mind to me is that AGI labs have various other triggers for flagging transcripts as requiring human review, which are looking for other types of information that would be of interest to them. AI-related capability insights would be my primary guess and concern, but other salient possibilities are also blackmail material, software exploits, and, well, math research results. They wouldn't be using that willy-nilly, because legal and reputational risks are still enormous while benefits are miniscule in nearly all cases, but if the stakes are high enough/the prize is really juicy (as in this case)... This seems entirely in line with what other major corporations have done in the past. If that feels impossibly beyond the pale, it's because, for v

For Ants and Anthropic-stans, does it seem like it would be good for the world if Anthropic implemented this proposal (suggested by Paul Christiano back in 2018) for implementing the ability to make trustworthy statements? If not, what seem to you like the main drawbacks?