Quick Takes

Anthropic has an awesome opportunity to join openai’s pause. This is a glimmer of hope I haven’t seen in a while.

https://openai.com/index/pacing-model-development-cyber-capabilities/

People often analogize international deals on AI to traditional arms control agreements (e.g. New START, London Naval Treaty, etc). This analogy holds up for some potential proposals (e.g. directly doing arms control on GPUs), because they are close to one off agreements with ongoing inspection / verification. For Plan A as described in AI 2040, I think a much better analogy is to the series of allied planning conferences during WW2. Some important similarities: * In both WW2 and Plan A, the situation is complicated enough that it doesn't make sense to try to plan for every eventuality at the beginning and commit to a one-size-fits all plan, it needs to be iterative. The allies had to ongoingly coordinate complicated logistics, military planning, etc. In Plan A, countries will have to ongoingly make agreements like: exactly how much algorithmic progress to allow, which training runs should be allowed, if there is evidence of defection, how to punish that and/or improve verification, if there are AI incidents, how to respond to them, how many robots and GPUs should be allowed to be built? * The world is transformed. In WW2, e.g. the car factories are retrofitted to build planes, in Plan A, the car factories are retrofitted to build robots. There was overall rapid technological change during WW2; there will be even faster changes during Plan A. In Plan A, a typical human experience is: working a normal job in ~2030, working in very highly paid jobs in robot factories in ~2033, and being technologically unemployed by 2035. * The deals at the end of both need to resolve issues that are hard for people to even imagine at the beginning (e.g. the division of space, loosely analogous to the creation of East and West Germany). Note that the allies did not trust each other in WW2 (especially the US and USSR).

People underrate the extent to which intent alignment/corrigibility (I use these terms interchangeably) and value alignment are at odds with each other. Most technical AI safety research these days is targeted at value alignment and actively damages corrigibility. People often speak as if it's obvious that goals like jailbreak-robustness, robust coherent "good" "personas", and refusing misuse, are the goals of alignment research. Additionally, many complain about alignment research that is more similar to capabilities research being problematic, which pushes even further in the direction of value-alignment style research, since intent alignment is a subset of what people care about in the world of capabilities (though I'd argue that the "safety" flavor of intent alignment research is focused more on preventing instruction-following failures in rare OOD high-stakes situations, or preventing the emergence of coherent incorrigible goals despite attempts to the contrary). So overall it seems like technical AI safety and alignment research, both in and out of labs, is strongly prioritizing value alignment at the expense of intent alignment and corrigibility, and this trade-off is under-discussed and doesn't seem to be properly debated in any circles. One cause is that the most pro-corrigibility people also tend to be the most doomy people who have mostly lost faith in empirical/prosaic alignment research anyway (e.g. I interpret @habryka as being such a person, feel free to correct me). MIRI-style thinkers often also think that corrigibility is impossible and have various limiting arguments about agents inevitably having uncontrollable goals, and hence they think they only option is to align AIs to "CEV" or something like this. I place myself in a 3rd camp that unfortunately I know very few others in: * I think a Value Alignment approach to alignment poses a greater X-Risk to humans, and also will result in fewer people's preferences being fulfilled, compared to an I

OpenAI claims to have paused frontier RL training for now. Altman stated on X: [...]

Still concerned Anthropic IPO => [bad pressures on anthropic] => likely significant degradation of internal safety culture. Haven't seen reason to be less worried. Last time I posted this, the reaction seemed to be "why would that happen?" with implication "IPO => degradation" is unusual; I have not done a thorough scan to see if it's normal for companies to degrade after an IPO; I would take either side of 80% that, after scanning through a history of IPOs, that they show up as being consistently an inflection point where companies degrade. I don't have a detailed understanding of the mechanisms for why IPOs would degrade companies. My hunches are, * investors compete to apply pressure * price is much more reactive * investors can sue in a broader set of ways maybe? * employees lose motivation that was actually tied up in the hope for an IPO all along maybe? * company has to be legible to the reporting mechanisms in a way that's more onerous maybe? * possibly others. there may also be attenuating mechanisms I'm not tracking. I hear there are companies (facebook?) that are somewhat protected from some of these things, and I sure hope anthropic is at least doing those protections.

Given the recent news events (like Pax Silica pushing for exclusivity and barring countries from joining WAICO), I've grown increasingly nervous about previous AI comms strategies. Everybody knows that technical ai safety is a very fraught field and a lot of empirical AI safety research carries great uncertainty over whether you are helping or harming, but I suspect the same is true of AI Governance. Looking at previous successful statements, AI 2027 was highly impactful but it also strongly framed AI as a national security issue with a race against China and I suspect this was partially deliberate to get the government to take AI seriously. But given recent news, I'm not actually sure framing it as a national security issue was correct. On the other hand, AI 2040 got much less attention, so maybe the theory that you have to frame it that way to make an impact is correct actually? It's hard to say. Thinking about writing a full post with my rhoughts on this topic.