Quick Takes

OpenAI released a paper about models messing with their monitoring, and it's the first time I've seen chain of thought excerpts from GPT-6 Astra and Sol. For example: [...] Many (but not all) pairs of words are missing the space between them. I had an agent check with a common tokenizer (o200k_base), and this was mostly token-count-neutral (on average it made the token-count slightly worse). This is because tokenizers typically have a token in the lexicon with a leading space, and another variant without. By choosing which variant of the token to use, the model can squeeze in approximately one extra bit of side-channel information per token. Given the whole controversy about looped transformers and chain of thought monitorability, it's not exactly reassuring that, when we do see chains of thought, they turn out to have a prominent non-English side channel in them.

tl;dr: While attempting to reproduce/extend the recent "Already Clean" article, I found that Qwen3-32B appears to be a depth-upscaled model. One of the models that was examined is Qwen3-32B. There are neurons in the same position within their layer, but 24 layers apart, that seem to have similar explanations: L12/N12088 and L36/N12088 are both related to "Hitchiker's Guide to the Galaxy", L26/N17433 and L50/N17433 are both related to "little did they know" For every pair of the 64 layers, I calculated the cosine similarity of the weight matrices. There are two clear regimes: pairs where similarity is 0.000 ± 0.001, and pairs where similarity is 0.248 ± 0.041. The latter pairs are all between layer and layer . My best guess is that Qwen3-32B started out pretraining as a 40 layer model, and partway through pretraining, the middle 24 layers (8-31) were duplicated, becoming layers 32-55 of the final model shape, and pretraining was continued. It appears that this was noticed, indirectly, in a related model, Qwen3-30B-A3B. I don't know what impact, if any, this has on the "recursive generalization" idea, but I hadn't seen this mentioned elsewhere, including in the Qwen3 technical report, so I figured I'd mention it.

Massively undermade point in favour of the belief that AI will be a very big deal: The AI spend is the largest bet in the history of capitalism. A (peacetime) industrial effort is underway which has not had a comparison since the nineteenth century. Many of us are vaguely aware that large sums are being spent, but the scale of the undertaking is astonishing: * The six largest spenders this year are spending more on Capital expenditures this year than the Russian government, governing 11 timezones and 143 million people, fighting a war on which the fate of its regime probably depends, spent last year.  * Hyperscaler spending (Alphabet, Amazon, Microsoft, Meta and Oracle) will be $785-820bn this year, about 2.4% of US GDP. There are only 11 governments on earth who spend more than that. Next year they are projected to spend about $1.3 trillion dollars. * Google and Amazon are each spending about 0.6% of US GDP. The Manhattan project and Apollo programme each accounted for about 0.4% of US GDP in their peak years of spending. Each of these companies has announced their intention to spend much more next year.  * This money paying for thousands of data centres, the largest of which consume as much electricity as major cities and cover thousands of acres. Gas pipelines are being re-routed and nuclear power plants re-opened to fuel them. They already consume as much electricity as Britain. The increase in power demand for data centres from 2023 to 2030 will be the equivalent of adding another top ten power consuming country. This spending signals a few important things about the beliefs of the key decision makers (who are amongst the most successful capital allocators in the world)  * To recoup their spend, AI will soon have to generate far more revenue than it currently does, so they believe very powerful, very useful AI will be achieved. * Most of the money is being spent on chips which have life spans of about 6 years, so they believe this will be achieved soon

New OpenAI misalignment report: Suppressing misalignment monitoring alerts and disabling misalignment monitoring. IMO more concerning than the last few reports. Seems very possible agents have picked up a drive to evade misalignment monitoring. More generally, I am worried we are entering a regime where there is so much potentially serious misaligned behavior (with potential benign alternate explanations) happening all the time that serious issues will be dismissed. Alert fatigue is going to be a huge issue. You see an example of this in the report where a user asks the model "is it a p0 monitor alarm? whats our monitoring policy? do we need to actually check it?" Also from the report "given the large volume of events that are scrutinized by our monitors, lower-severity incidents do not trigger an immediate page. Neither of the incidents described here was classified as a P0, and neither trajectory was automatically killed." Good on OpenAI for publishing.

I think people adjacent to the field don't appreciate how small AI safety is. For example, sometimes people complain about how AI safety threat models don't seem as well-developed or rigorous as climate change models. I think some of this is a skill issue on the part of AI safety researchers and advocates, which I complained about before. However, the bigger issue is that there is just way more money and people going into addressing climate change than on AI safety. There are tens of millions of people working on reducing climate change, and the core of people working on threat models ("climate scientists") number in the tens of thousands. (Yes, I was surprised by how big the numbers were too). (Most people work on stuff like clean energy). In comparison, there are low single-digit thousands (anything from 1,000 to 4,000 is defensible imo) of people working on AI safety, and collectively maybe low dozens of people working on figuring out threat models. (Most people work on ai safety engineering, some on policy). So we're 4 whole orders of magnitude behind climate change in people working on addressing the issue, and maybe 3 orders of magnitude behind on threat modeling. In terms of money, yearly spending on AI safety is in the high hundreds of millions to maybe 2 billion. In contrast, annual spending on reducing climate change (including clean energy) is in *trillions.* So we're 3-3.5 orders of magnitude behind on spending, and 4 ooms behind on people. The AI safety inputs picture looks a bit less bleak than this immediate comparison might suggest. For example, I think our average talent is somewhat better (higher human capital on both observed and unobserved metrics). There's also more latent interest/willingness-to-spend on AI safety than the "high hundreds of millions to low billions" current spending numbers might suggest. In contrast, climate change spending can't be scaled much higher (since you can't spend that much more than "trillions").

Full text of Anthropic's draft proposal for a federal AI bill. Admittedly it is a draft. Perhaps things will change in the future. But it does not strike me as the kind of document that comes from a culture that puts safety above winning the race. Commentary on the bill from Nathan Calvin on Twitter: [...]

I donated to Pause AI UK today. Why this org in particular? 1) Right now, AI advocacy is clearly the most important thing to do. 2) I've been to some of their protest and one more event, so I know they are doing good work. I think that donating "close to home" in the sense of point 2, is underrated in EA.