Quick Takes

o3 has been used for nuclear weapons research 1. Vox reports that o3 weights have been transported in a suitcase to an air-gapped supercomputer to help with nuclear weapons research. 2. This has seemingly gone unnoticed in the broader AI safety community. Although I do not know how to update based on Vox's article, it seems like information that more people should know because it is important information about the integration of (frontier) AI into the US military. At least some of the people who work there do not seem concerned with catastrophic AI risk. Relevant quotes from when the journalist Joshua Keating was there in January this year: [...] AI risk does not have a good reputation it seems: [...] Side note: it seems odd to imply the motivation for the Manhattan project was not started as a race. Here are further quotes that I found interesting (emphasis mine) This should allow you to get most of the information without reading the full article. [...] People seem to like it: [...] More integration is coming up: [...] What the supercomputer does: [...] What o3 might actually be working on: [...] Tool-framing of AI: [...] Although the majority of the budget is for weapons research, other research happens too: [...] More AI is coming to the US military: [...] Thanks to Bronson Schoen for sharing the article with me, and Alex Lloyd for feedback.

Shower thought I had: [...] For example I've noted before that I suspect current levels of AI persuasion/manipulation capabilities has lagged behind AI capabilities overall (this is nonobvious and I don't defend it here but my impression from talking to empirical researchers on AI persuasion is that they broadly agree with me). Now imo this is probably on net a good thing, though it carries with it real risk (of rapid catchup growth). Like personally, at any given capabilities level I'd be happy that the models are worse at manipulating humans! But I could imagine that ppl who are more into overhang-style arguments would rather the persuasion capabilities develop smoothly on a predictable curve, so society has time to incrementally respond for every new generation.

Possible test to check whether X is your crux for believing Y: ask yourself, if you found out X was false, would it make you say "holy fuck, I must be wrong about Y!!"? That is: would you happily admit that Y must be false too? Would you feel glad you noticed X was false, because it set you straight about Y? Example where the answer is "yes" for me: I think physics is a serious field of study with reasonable standards for publishing. As a consequence, I think a non-physicist wouldn't be able to get a parody paper published in a reputable physics journal. But if someone succeeded in doing that, I would go "holy fuck, I must have been wrong about the field!!". — If instead the prospect of being wrong about X makes you feel angry, worried, or defensive about Y, then maybe that's because you're imagining that you'd still believe Y, and that you'd see X turning out false as just an infuriating coincidence that other people will use an excuse to doubt Y. If that's how you'd feel, then X probably isn't a crux for you. Example where that's how I'd feel: I also think that physicists would probably fare better than laypeople at learning Chess. That is, if you take a bunch of physicists who don't play the game at all and a bunch of laypeople who don't play it either, and gave them each three months to get good, I think the physicists would end up stronger. That's because I think physicists are just generally smart and serious people, and I think being smart and serious are huge boons for learning Chess quickly. But if someone did this experiment and I turned out to be wrong, I would roll my eyes and feel hurt about it. It would make me want to say "I guess Chess is a stupid game that you can get good at by being stupid just as well as by being smart". I wouldn't feel like physics had been debunked; I'd feel like it had been attacked. Not a crux. Side note: I'm a little unsure how I feel about that last example, but I'm posting anyway because Quick Take.

There's a massive divide between the Consensus and the Slighted that runs through many domains. I think healing this would in particular help decrease AGI X-risk by decreasing the motivation to make AGI and decreasing the motivated reasoning around X-risks. A conjecture is that both poles are attracting states in social space (let's say the vector space of people represented as a vector v where v_i is how much you talk with / listen to person number i).

Barry Mazur on the idea of a "critical mass learned → phase shift" in one's view of a subject: [...] Somewhat relatedly, Venkatesh Rao on how reading a lot phase-shifts you as a reader: [...] I think this is necessary, or at least helps a lot, with reading between the lines, whether in philosophy or fiction or such, otherwise you think it's all just incoherent/incomprehensible/pretentious BS. (Of course it may still be.)

Our OpenAI alignment blog is updated pretty frequently these days with research results that should be of interest for folks here https://alignment.openai.com

AI safety people often worry about their work having capabilities externalities. I feel like the inverse problem is often a much bigger deal: the stuff you do might be done later by the capabilities/product people in the course of their work, meaning your work isn't counterfactual! I think people should try to distinguish these concerns more clearly. (See also Bronson Schoen's similar post three hours earlier. I wrote this shortform in Redwood slack a few weeks ago and am just posting it now. Obviously this isn't a very original take.)