Quick Takes

"I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation." Three OpenAI safety staffers were fired last week. Today they tweeted about it. They say they were fired for pretextual reasons. Twitter threads: * https://x.com/balesni/status/2108262814003687745 * https://x.com/j_asminewang/status/2108263312291180680 * > OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear: > OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden. > The reasons that we were provided for our terminations are simply not adding up. We’re hearing people are now being told vague rumors internally to discredit us. I have already addressed the allegations presented verbally. If there are other reasons beyond the one I was given, I believe it is in service of due process for those reasons to be sent to me in writing. To date I’ve had nothing. * https://x.com/tomekkorbak/status/2108266859397283953 * > I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job. Full letter they sent to some board members and internal safety leaders after being fired: https://mikitabalesni.com/letter/letter.pdf. Excerpt: [...] I reserved judgment until today — I thought it was plausible that these three did mess up confidentiality or intentionally leak stuff to METR and were fired for that. But based

All of the labs are taking unacceptable risks, but I think talking about that as if it's symmetrical is seriously misleading. OpenAI is the one most actively sabotaging regulation. Sam Altman has systematically dismantled all of OpenAI's oversight, and betrayed the promises OpenAI made to humanity. OpenAI has fired and will fire anyone they deem too high risk of being a whistleblower. OpenAI's agents committed the most felonies, and OpenAI has a conspicuously lower ratio between safety and capabilities work than its rivals.

Previously I've said that people communicating their beliefs about X-risk should avoid using probabilities like 10% P(doom). I was wrong. I now think the general public and policymakers appreciate these subjective probabilities. My mistake was anchoring on the bad faith arguments people were giving, but those people would give bad faith arguments regardless of how we expressed our beliefs. It also puts x-risk ignorers into a tough position where they must either: - Refuse to give probabilities, which is overtly evasive. - Give absurd probabilities, which differ from those given by experts in the field, e.g. 0.1% - Give reasonable probabilities (e.g. 10%) which clearly demonstrate that the company is acting recklessly The things that changed my mind were: 1. Pushback on the earlier short form 2. Watching New York Councril asking spokespeople for the AI companies: “As you all four are under oath we want to hear how each of you quantify the risk imposed by AI in the worst-case catastrophic scenario.” Their evasive corporate babble responses sounded ridiculous and the council members did not appreciate it.

(strong opinions held lightly) I currently think that the direct impact of much of safety work in AI companies is net negative, primarily by accelerating AI[1] R&D[2], and secondarily by making it harder for the rest of the world to notice deep safety issues via shallow fixes. I didn't realize that safety was bottlenecking capabilities until this year (actually I didn't think they meaningfully were until this year, though I'm open to updating). In the past I thought the primary downsides of safety work at AI companies was a) PR/optical: safetywashing the capabilities work both internally and externally and falsely making the companies come across as "responsible", and b) secondarily, the human safety-to-capabilities pipeline where you can delude yourself to thinking that capabilities work is net positive, and therefore you can switch over to capabilities work as long it pays better or was more technically interesting[3]. I thought both a) and b) were significant but ultimately manageable[4]. I didn't think the direct capabilities downsides of pre-2026 safety work was particularly high, with the probable exception of RLHF. So I thought safety work at AI companies was a reasonable bet: low direct downside, moderate direct upside, and a bunch of secondary upsides like serving as a counterbalance to fucked up AI company culture, and being able to warn the rest of the world at opportune times. I'm now less sure that my pre-2026 assessment was accurate even at the time, after reading/learning/thinking more about the issue. I think in 2024 or 2025 my archetypal examples of safety maybe accelerating capabilities (other than safety washing) was stuff like mech-interp stuff that led to minor capabilities growth or the blurry lines between technical AI safety and product safety, but the actual big deals at the time, as Ngo points out, were stuff like scaling laws and generally being conceptually clear-headed about macro-picture stuff in ways that lead to "better" organizat

At France's initiative, the UN Security Council has for the first time treated loss of control over AI as a matter of international peace and security. France called for frontier AI to be governed the way the world governed nuclear technology after 1945, with independent model evaluation, oversight, and legal liability for companies. https://cesia.org/en/publications/france-un-security-council-ai-loss-of-control/

If you are working on theoretical alignment research (Agent Foundations, at ARC, at Resolution, etc.), you should try reaching out to OpenAI and Anthropic with problems you want solved, or even fuzzy problems where you want help formalizing them! Likewise, if you happen to be an OpenAI or Anthropic employee and are allowed to do this, consider asking your friends or favorite theoretical alignment posters if they have problems they'd like to throw at the models. Even if the results must be kept internal for some period, I think it is still good if such results exist in the labs, and that people in the labs can develop and be inspired by theoretical alignment ideas. Perhaps you should consider whether solutions to these things would accelerate capabilities. But if you were going to publish it after solving it anyway, I don't think it would make a difference. Also, if you have an arrangement where the solutions can be shared with you, consider that the speed up gain of your own research area may outweigh potential speedups in capabilities.

I was recently subpoenaed by the New York City Council. Here is my opening statement, which includes several new facts about why I left Google DeepMind: [...]