For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
For what it's worth, I think Daniel Ellsberg makes a pretty compelling argument that it would make sense for America to unilaterally destroy all their land-based ICBMs, but keep a few nuclear submarines to be able to credibly threaten destroying some major cities. This way, Russia would also have less incentive to maintain such a giant nuclear arsenal, as a major justification for having a large arsenal is that it is needed for taking out as many enemy ICBM sites as possible at the beginning of the war, to minimize damage to your side.
1If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win.
I'm still confused why people treat "solve alignment properly" (usually meaning that it works up to arbitrary levels of superintelligence) as the win condition to aim for.
Once you have an aligned superintelligence, you still need to use your superintelligence to make sure that other countries don't build misaligned superintelligences, or that small countries or terrorists don't build bioweapons in a way that the superintelligence also has a hard time protecting against in a way that is acceptable to the average voter (e.g. putting nanobots in everyone's body is something at least 30% of people won't consent to).
So the win condition is not building a superintelligence, but a country or preferably a coalition of countries having aligned and capable enough AIs that they can stop the global arms race and make sure no one else builds more powerful AIs or terrible weapons. But this win condition can plausibly be achieved at a much lower capability level than a superintelligence at the limits of intelligence.
So I think people should think of their goal in Plan B/C not as getting to the point where they can use their AIs to solve the (imo likely very hard, and definitely super hard to verify) problem of solving alignment up to arbitrarily high intelligence levels. Instead, they should aim for the (still very hard, but imo still a lot more tractable) goal of building AIs that are capable enough to disarm nations not joining the coalition, and aligned and trustworthy enough that one can trust them with a type of handover where they keep the global peace and make sure no one builds smarter AIs for a long time, night-watchman-state like. After that, the night-watchman-AI can let the humans figure out in the next thousand years what they want to do with the world, and whether they want to build an actual superintelligence once the thousand years run out.
See a more detailed argument here.
1
The AI Futures Project[1] distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories:
Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely.
Plan C (my combination of their Plan B/C+/C/sort of D)[2] is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to stop the race, that six-month or two-month lead is the amount of time they have to prepare, and they can spend it more or less wisely.[3] The point of burning their lead at the last moment is that they can do alignment research with very advanced, right-on-the-brink-of-superintelligence AIs. This is a dangerous plan, and there would be many subtle points they would have to get right, but in a world where nobody can get a global pause in time, it's hard to think of a better one.
Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty. Plan C benefits from alignment-pilling companies, having the alignment-pilled companies be as far in the lead as possible, making sure the alignment-pilled companies will burn their lead when the time comes, and giving them as good alignment teams as possible (so that burning the lead produces good results).
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C. From about 2023 onward, as the general population woke up to the implications of ChatGPT, we rightly concluded that the chance of getting Plan A was better than expected, and we should invest more resources into trying to make it work. During both periods, the natural strategy was to work as hard as we can to get Plan A, where chances of survival are highest, but to be prepared to execute Plan C competently as a backup if that didn't pan out.
Now I think the pendulum has swung too far. New members of the community are unaware that Plan C was ever considered a possibility, or think it's some kind of accelerationist plot to distract us from Plan A. Older members heap scorn on it and act like everybody knows it's been debunked. Most concerningly, people propose burning our chances of executing Plan C in exchange for infinitesimal increased chances of getting Plan A, or even for no increased chance of Plan A at all.
I'll lump these ideas together under the heading of "SURA" - the idea that the only ethical course is for people/companies/countries to Stop Unilaterally + Raise Awareness, with no plan for what happens if race dynamics continue anyway. I interpret the planks of SURA's platform as:
All of these planks are net negative in a world where we end up needing Plan C:
SURA's continuing popularity despite these downsides comes from a set of questionable assumptions:
I'll start by challenging these assumptions. Then I'll argue that, even if you disagree and think some of these assumptions are true, SURA is actually not just bad for Plan C, but bad even for your chances of getting Plan A. Then I'll argue that the current consensus in favor of SURA comes not from people rationally balancing these tradeoffs, but from people accidentally shoehorning themselves into a mass-politics-shaped container, which forbids subtlety and demands clear villains.
I. Questioning Assumptions Behind SURA
I.A: Is Plan C doomed? Is alignment doomed?
There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves". I'll try to avoid this by giving my own model first, then citing experts who agree with me.
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking; as they become smarter, we worry (without total certainty) that they will become more coherent and scheming in their power-seeking beyond just being reward-on-the-episode seekers. There will, at some point in the future, be a point where AIs become untrustworthy for alignment research, although this can be mitigated by things like control (ie trying to make sure that if they sandbagged alignment research, they would get caught) and deals (ie making it in the AIs' best interests to cooperate with us). We don't currently have a good sense of when that point is, and it's possible that it's far away, can be delayed indefinitely, or doesn't exist (this last would be surprising to me, but many things that have happened in alignment recently have surprised me, starting with the existence of pretraining at all).[5]
Daniel Kokotajlo described current alignment as "more like psychology than engineering", which I agree with. But psychology (and psychiatry) have many success stories! I think of Plan C alignment (ie the kind we can do without decades of preparation) as tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime. I don't claim this is safe or good or will definitely last forever, but I claim that - being totally agnostic as to where the treacherous turn happens - this could delay it for quite a while (relative to capabilities). If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win. If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (ie GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.
All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from. "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A. But a global pause is also not certain to happen, and in a world where we might not have any other chances, we should be working to protect and fortify this one.
The experts' opinions: My favorite source for these is the AI Futures Project. In this document, they discuss the chances of survival on various plans. Here are each team member's estimates for the three scenarios I'm lumping together as Plan C. The first scenario (C+) involves the entire United States uniting behind a Plan C like effort (but a global pause remains impossible because China won't cooperate). The second scenario (C) involves one leading company (eg Anthropic) uniting behind it. The third scenario (D) is the worst, where nobody has any lead and we just have to hope for the best.
Daniel Kokotajlo: 40% ----- 20% ----- 10%
Ryan Greenblatt: 60%----- 63% ----- 55%
Eli Lifland: 50%------45%--------32%
Thomas Larsen: 45%-------35%-------20%
Romeo Dean: 40%-------35%-------20%
I also asked the AI superforecaster FutureSearch; its own number was 44%.
These numbers are mostly in the 25% - 50% range, not the 0.00001% that most people's actions seem to imply. The average gain from moving from Plan D (poorly thought out default version of the plan) to Plan C+ (consciously trying to get it right) is about 20%pp.
The same is true of other experts. Buck Shlegeris (formerly of MIRI, now of Redwood Research) hasn't offered a specific number, but said that "AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes. Paul Christiano's last number was several years ago, but it was something like 20-50%.
All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that the chances of getting a pause treaty were slim. The two clear estimates I could get were Eli Lifland (he said 15%) and the AI superforecaster FutureSearch (it said 2-20%, depending on how/when I asked the question).
So:
📌📌All six of these forecasters[6] (the five AIFP people plus the AI) implicitly believed that if humanity survived, it was more likely to be because something in the Plan C supercategory (B/C+/C/D) worked than because a global pause worked. 📌📌
(I'm adding the emoji because it's the one sentence I most want you to take away from this entire essay - there's a risk that people will treat everything other than the pause as an afterthought, but having a good backup plan is potentially the most important part).[7]
To clarify - this doesn't mean that we should prefer Plan C worlds to Plan A worlds. Plan A worlds still hold higher chance of survival if we're in them. It just means that given the low chances of getting into a Plan A world, the quality of preparation for the backup plan really matters!
I.B: Is racing to build superintelligence deontologically evil?
If Plan C is important, then this suggests we ought to be trying to alignment-pill leading companies, or put alignment-pilled companies in the lead - rather than demand that these companies leave the race or burn their lead. But should there be a deontological prohibition on racing towards superintelligence, regardless of whether it has good consequences?
I'm too uncertain on what determines morality to say anything about it with confidence, but the analogy that comes to mind is consumer use of AI, especially by safety organizations.
If no consumer ever used AI products, then AI companies wouldn't make any money, they would go bankrupt, and we would not get dangerous superintelligence. Therefore, it seems like maybe you and I (and especially organizations working on AI safety) should boycott consumer AI products.
But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users I've encountered). I think this is defensible. If you boycotted AI, it wouldn't make a real difference in the world (because you're just one person, and millions of other people would still use it). But it would make you less effective, and AI safety organizations would be able to do a less good job at their important work. Therefore, AI safety organizations should continue using AI, even though in a sense this causes the AI race to continue, because they're unable to coordinate a global boycott, and they understand that a unilateral boycott would have greater costs than benefits.
But it seems like this same argument should apply equally well to the AI companies, who also can't coordinate a global boycott simply by stopping themselves, and who also understand that a unilateral boycott would have greater costs than benefits.[8]
I.C: If lots of alignment-pilled people and companies quit, would that raise awareness enough to make a global pause much more likely?
If the deontological argument for SURA is "you're prohibited from racing", the primary consequentialist argument is that ceasing to race would raise awareness and make a pause more likely.
I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[9].
My very preliminary model would be that this halves our chance of getting that 20%pp bonus of going from Plan D -> Plan C+ (and so costs us 10%pp). It gets us an extra 3%pp of pause likelihood, but pause doesn't save us with certainty, so let's round this down to an extra 2%pp of survival, for a net cost of 8%pp of survival probability.
I.D: If alignment-pilled people and companies stay in the race, does that burn timeline?
A secondary consequentialist argument for SURA is that each person, company, and country that quits the race will gain us extra timeline to do good things (like work for a pause).
In some sense, this is definitely true - all else being equal, a race with more participants will be faster than a race with fewer. But I find this less impressive than other people seem to, for three reasons:
The magnitude is modest
I asked FutureSearch to estimate the Epoch Capabilities Index of the top model on 12/31/2030, depending on whether Anthropic quit the race today. It said that Anthropic leaving the race decreased 2030 capabilities from 230 -> 222. This is equivalent to their absence slowing things down by six months during just over four years, or 12%.
Subjective timeline ≠ objective timeline
Suppose that none of DeepMind, OpenAI, or Anthropic were founded, and the transformer was not invented, for fifty years - so all of this is happening in the late 2060s and early 2070s. Do we gain anything from this fifty years of extra timeline?
I don't think so. We didn't accomplish very much of interest in the early 2010s. The alignment work being done at the time was agent foundations research that stagnated for lack of broader interest; policy work was good but nobody listened to it. Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.
This same lesson - an extra year of an earlier situation doesn't buy us as much as an extra year of a later situation - remains true today. A month of the pre-Coxon situation buys us less than a month of the post-Coxon situation. A month without automated-mechanistic-interpretability-researchers buys us less than a month with them.
People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me. As such, I discount the six extra months we could get from having Anthropic leave the race by ~half, because my subjective impression of the value of six extra months is partly that it produces AI progress that contributes to wakeup.
The whole point of timeline is to burn it on good things
If someone proposed spending the next six months on some kind of plan that would make the leading companies become much more alignment-pilled, this would be a better use of our time than any actual plan I know about.[10]
This suggests we shouldn't be trying to kick alignment-pilled companies out of the race to save six months.
II. Actually, most of this stuff is bad for a global pause, too.
I've been framing this as sort of "Against Things That Trade Away Likelihood Of Succeeding At Plan C For Merely-Very-Slightly-Higher Chances Of A Global Pause". But even this is a concession that relies on assumptions I don't hold.
The key demand of SURA is that alignment-pilled people and companies stop unilaterally. The likely end result is that we only have non-alignment-pilled companies, staffed entirely by non-alignment-pilled employees, left in the AI industry. [11]
But the presence of alignment-pilled companies is one of the strongest levers that we have to change the political situation and enact a global pause. Eliminating them makes it harder to win political fights, convince the masses, and convince China.
II.A: Getting rid of alignment-pilled people and companies makes it harder to win political fights
The most prominent AI safety political fight of the past year, the attempt to elect Alex Bores, was fought primarily with AI money. Of pro-Bores spending, about 60-80% came from Anthropic and its employees; of anti-Bores spending, about 95% came from an anti-tech-regulation coalition headlined by OpenAI co-founder Greg Brockman. If this is a prelude for the fights to come, then even the slightly different level of social responsibility shown by Anthropic vs. OpenAI here matters a lot.
Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.
(Greg Brockman recently stopped funding Leading The Future, supposedly after pressure from pro-regulation OpenAI employees, but I think this just highlights my point)
II.B: Getting rid of alignment-pilled people and companies makes it harder to activate the masses
But suppose you believe that Bores-style incremental regulation won't matter anyway - what we really need is for the public and policy-makers to spontaneously rise up and demand a pause.
I've been monitoring how that's been going, and so far the three awareness-raising things I've been most excited about are the Pacing The Frontier letter/plan, the Hugging Face incident, and the Jacob Coxon resignation.
In the Pacing The Frontier letter, OpenAI and Anthropic as corporations, and 1,386 of their employees as individuals, all endorsed a joint statement requesting that "the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." (Dario Amodei signed; Sam Altman's signature was notably missing). I don't know how the people trying to get lab employees to quit think of this. Was this 1,386 failures? Would they have preferred a world where all of these people quit the day before, and letters like this never got written?
In the Hugging Face incident, a swarm of rogue AIs hacked the site Hugging Face, raising awareness that misalignment was not just a possibility but a reality. Here, OpenAI gets maybe a D+ for transparency. Once Hugging Face caught that someone was hacking them, OpenAI came clean that it was them. And they let METR and Redwood in to investigate for a few days, even though this wasn't nearly enough time. But even this D+ in transparency radically transformed the discourse beyond what an F would have done - if they hadn't given METR/Redwood those few days, we might never have heard about PHASEONE[big] and all the rest. We already know from the great work by the team at Collusion.wiki that OpenAI is sitting on more bombshell revelations. The difference between a 10th percentile company here (would have tried to hide the whole thing, or disguise it behind a one line statement, and wouldn't have let METR/Redwood in at all) and a 90th percentile (would have given us the full deal, including things we don't know about now) would have been transformative to the discourse. So if, like me, you think that the level of mass political mobilization depends on how many warning shots we get, you should be thinking very hard about how to maximize transparency and therefore warning shot number.
Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies. But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight. When people tried to tar Jacob as some kind of crazy, it was an outpouring of support from other non-resigned people at Anthropic - especially Evan Hubinger, who said he also believed in existential risk - that went viral and convinced people that Jacob was for real. If Coxon had never joined Anthropic to begin with, or if everyone like Evan had quit, this couldn't have happened.[12]
II.C: Unilateral disarmament makes it harder to bring China to the bargaining table.
In March, I wrote a snarky post called Every Debate On Pausing AI. It made fun of our misinformed opponents, who insisted that if we unilaterally paused AI then China would win. But actually, I said, nobody wants to unilaterally pause AI; this is misinformation. All reputable AI pause groups want a bilateral treaty that pauses both sides at the same time.
Here I am forced to eat crow; as things shift ever more towards SURA, there are increasing demands for the US to pause unilaterally. Bernie Sanders' "Ban Artificial Superintelligence Act", which was endorsed by MIRI, is approximately the following plan:
But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
I've heard people say that maybe this would shock China so much that they would realize we were honest in our fear of superintelligence, and change their minds. I don't dismiss this possibility, but it also seems possible that China - having seen us place overly burdensome restrictions on nuclear power, infrastructure, and housing because we are a "lawyer state" vetocracy - would not be very shocked to see the US' leading socialist place restrictions on AI too, and would continue pursuing AI for what they think it's good for - a dual-use technology which could give them a major although not world-changing military and economic advantage.
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[14]
(Maybe you already care a lot how alignment-pilled companies are)
An expansion on the above: we get angry when AI companies do bad, non-alignment-pilled things. When Anthropic accidentally trained against chain-of-thought, we got mad. When OpenAI non-accidentally implemented a precursor to neuralese recurrence, we got mad. When Anthropic wrote a report on misalignment to Congress which soft-pedaled various alignment failures to the point of misinformation, we got mad - and I saw some people call for Anthropic to fire the perpetrator.
I agree: let's have Anthropic fire people who do bad, non-alignment-pilled things. But then what? Anthropic replaces this politics coordinator with a great, alignment-pilled politics coordinator who really cares a lot about getting things right . . . and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone. How is that helpful?
If you can be bad, you can also be good. We don't hesitate to notice and protest when AI companies are much worse than even our low expectations, but this itself demonstrates that we care a lot about whether labs are at the 10th percentile vs. 90th percentile of good alignment-pilled behavior.
III. What incentives are we fighting here?
A common argument for SURA is that even if corporations could theoretically be good, we cannot trust actual corporations (or corporate employees) to behave this way; the lure of money and power is too strong.
I think this is a good directional argument for why companies need to be careful, but that actually, everything is corrupting, there is no option where we avoid touching potentially-corrupting things, and instead of treating corporations as corrupting and everything else as safe, we should be working on the general problem of operating in a dangerous and corrupting environment, which we can then apply to corporations as well as everything else.
III.A: Are corporate incentives a "One Ring" that corrupts everyone who touches them?
Suppose someone claimed you could trust them with the One Ring. You could trust them because they were only doing it to keep the Ring away from other, worse people, thus solving the problem of there being a Ring at all. Sure, they had a complicated history of sometimes using the Ring's powers when it was, you know, important. But those were, like, emergencies, and they promise not to do it again, except in, you know, more emergencies. Besides, if you don't give it to them, then the worse people will get it, and it's not like you can just leave it lying around or something.
The person who claimed this was named Frodo Baggins, and he saved the world.[15] There are no simple answers to the question of how to balance the corrupting influence of power vs. the fact that if you never take power you will be powerless. But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.
I question the assumption that good people dare not touch the concentrations of power at frontier AI companies because they will be corrupted before they can do good work. History is full of examples of people at these companies deciding they didn't like something, then quitting or changing it at great cost to themselves. Dario and the other six Anthropic co-founders quit OpenAI to start Anthropic, even though it cost them their unvested equity in a company that was already worth $5-15 billion, and even though they originally assumed Anthropic would be a research-focused company that might not make any revenue (really!) Paul Christiano left OpenAI to start ARC. Ilya Sutskever was nervous about Sam Altman, so he risked blowing up the company he co-founded to get him out. A few dozen people have resigned once they thought they could no longer participate in the company ethically, most famously Daniel Kokotajlo, Jacob Coxon, and Alex Turner, but also a bunch of people we don't know about whose concerns seem almost trivial in retrospect - Suchir Balaji quit OpenAI over copyright issues, Blake Lemoine got himself fired from Google by refusing to stop arguing that their chatbot was sentient. "OpenAI's safety team quits en masse once per year" has become a meme.
The people I know who joined AI companies seem like the same people they were before. Some of them were never that concerned about x-risk, and continue not to be concerned. Others were always very concerned, and remain concerned. When they've changed positions, it's been because their factual models have changed, not their personalities.
We aren't the first people to face these questions. Anyone who cares about any cause has had to decide whether to pursue it as a lone voice crying in the wilderness, or as a part of some powerful organization. Some people run for office (government is hardly known for not corrupting its participants!) Others become activists in a mass movement (activism is hardly known for being a paradise of epistemic virtue!) Still others join big NGOs or academia (...) A few start companies, trying to prove they can do things the right way. Most progress has come from people like Frodo who understood the risks of these paths but did them anyway.
III.B: I dunno, mass politics is also looking kind of annular
A background assumption of the "corporate incentives are too corrupting to touch" claim is that working in mass politics is a safer and less-corrupting option. Given famous LessWrong posts like Politics Is The Mind-Killer, I think we should be more skeptical of this assumption.
The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.
This has some resemblance to SURA, and I think this is more than coincidental. Convergent evolution is trying to carcinize us into yet another political-movement-shaped crab. SURA-as-politics is turning into a "superweapon", bristling with its own lexicon that makes dissent hard to express:
All of this produces epistemic closure - if someone disagrees with SURA, or worries too much about race dynamics, then they've either been corrupted by the companies, or are "deferring to" "EA".
It looks like what one would expect if mass politics were trying to taking AI safety on the same path as climate activism or socialism. The clearest analogy is to false consciousness - the socialists' idea that everyone who doesn't currently agree with them has either been corrupted by their proximity to money/power, or at least brainwashed by a hegemonic capitalist system.
See also this article in a socialist magazine about how every tiny socialist party believes that if they could only knock out their true enemies - all the other tiny socialist parties - then they could get a monopoly on socialism and overthrowing capitalism would be easy. Cf. SURA as the belief that if we could only knock out our true enemy - Anthropic, effective altruists, etc, then defeating misaligned AI would happen naturally. For example, why do we criticize Anthropic more than every other AI company combined?
(Exhibit 1: Sigal Samuel's "Anthropic Could Just Stop")
Most recently, I've seen people in the AI safety community reblogging and pointing to Sigal Samuel's article Anthropic Could Just Stop (seen elsewhere under the title "Anthropic Should Just Walk Away From The AI Race"). I think it's a good example of what I'm complaining about - something which uses the right shibboleths, makes it sound like race dynamics have been solved and we don't have to worry about them - but ultimately has no answer to the race dynamics which are the whole point of worrying about this question.
Here are some sample phrases from the article:
This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".
But where is its actual argument that race dynamics don't matter? Or that Anthropic leaving the race won't end up with OpenAI / Meta / China continuing to race and creating an unaligned superintelligence? The only place I can find where Samuel attempts to address the possibility is this section:
It starts by framing this in a maximally hostile way ("a perennial objection...'but China!'), letting us know we should dismiss it as an enemy soldier. Then it says that "AI is not just one thing with one purpose...so what matters isn't simply who gets it first, but who uses it in which ways". I don't understand how this is supposed to relate to the worry that China will remain in the race if Anthropic leaves, especially the fear that nobody will be able to "use it" at all because it will be misaligned.
Next, it says that since the harms from AI are massive, everyone except Anthropic might just stop on their own if Anthropic stops first. Big if true!
(again, the AI superforecaster assigns this a 3% chance)
This is just hand-waving away the potential that there might still be race dynamics after all the good guys drop out. But the whole point of the good guys not immediately dropping out is the belief that the race dynamics will continue. Indeed, the race dynamics continuing ought to be the very-high-prior default hypothesis, since everyone else has said they're not going to drop out, and since other companies dropping out seems strictly less likely than Anthropic doing so.[17] As such, even though this article is very long and detailed, it feels like a cat-belling problem, where it addresses every objection except the central one that was the only reason anyone ever disagreed in the first place.
I don't mean to attack Samuel too hard here. She is faithfully describing the position as best she can; I think the position itself struggles to answer this question.
(A virtuous exception)
I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
This makes sense, and I exempt it from my criticism.
IV. What if we survive?
I don't think this should be our main consideration - we should start with trying to minimize the odds of extinction - but I still think it's important.
If we survive x-risk, the next battle is concentration-of-power risks. It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world. They'll face a choice between possible actions including:
It's all nice and well to say that we should never let ourselves get in a position where anyone has all of these options, and that national governments should form a plan for what to do before this - not even the real national government led by its current leaders, but some ideal corruption-free national government which represents all of our best impulses. But, much like a global pause, this is a great outcome which we might not be able to prepare in time. If we get ASI next year, the ideal perfect government might be nowhere in sight; we might face a situation in which Trump demands absolute power, and the leading AI company has to decide whether to give it to him. Or one where either the USA or China will get control of the entire future, and which one it is depends on which AI company is in the lead. Or one where democracy miraculously survives, but the AI company and its investors end up with 90% of all wealth, and they have to decide whether to redistribute it vs. fight to keep it (using superintelligent lobbyists that give them significant control over the democratic government.) If any of this happens, the shape of the future might depend on who the leading AI companies are.
There is a certain Far Mode sense in which believing that Dario Amodei and Demis Hassabis are exactly as untrustworthy as Elon Musk and Xi Jinping helps reminds us of universal human fallibility. But do we really want to bet the world on it being literally true?
I think that a significant percent of people, put in a position of absolute power, would try to do something good, like make sure AI follows some kind of democratic rule of law. And some other significant percent of people would try to do something bad, like become a dictator. Saying "we should never let anyone get into this situation" isn't a solution, it's a wish - it may not come true, and we shouldn't be pursuing plans that will fail catastrophically if it doesn't.
SURA effectively removes all ethical people from the bargaining table in situations like these, ensuring that - unless the virtuous government representing our best impulses really does appear and takes control of the situation - the future of humanity will be handed over to sociopaths and cowards.
V. What's The Alternative To SURA?
The alternative to SURA isn't abandoning awareness-raising, mass activism, and support for a global pause. All of these strategies are great and important. It's viewing them as part of a portfolio of different options, none of which need to be set on fire to boost the others.
Instead of spending our corporate activism energy telling companies to stop unilaterally, I think we should spend it:
V.A: Getting companies to agree to stop in a coordinated manner
By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.
This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.
This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened. Also, I think SURA got to the STAIR people too, and now they're demanding labs pause unilaterally instead.
We should be trying to figure out what a binding version of this commitment would look like. Additionally, we should be spending more energy figuring out what the antitrust barriers to such a statement are, how to get around them, and what demands we want to make for companies to violate antitrust law (it seems plausible that we should demand they do this and let the administration pay the PR cost of trying to stop them, but this is an unconsidered opinion and it might be that there are better options).
(...and then we know where we stand)
If all American, but no Chinese companies are willing to sign the commitment, then we know that we have a diplomatic problem, and can concentrate on diplomacy.
If all Chinese, but no American companies are willing to sign the commitment, then we know that we have a domestic problem, and can concentrate on domestic legislation, encouraging policymakers by letting them know that if they force US companies to comply, there are no further concerns about international race dynamics.
If everyone except one company is willing to sign the commitment, then we can start putting pressure on that one company, boycotting their products, and applying as much social sanction as we can to their executives and employees.
If all leading companies, but no trailing companies, are willing to sign the commitment, and we can't change this, then we know that we're in the right position to focus on Plan C+ or Plan C.
V.B: Getting companies to prepare for Plan C+/Plan C if that doesn't work
Although we should hope for the best, we should assume that not all companies will sign a conditional pause commitment, and prepare for what to do.
In my modal non-hopeless play-to-our-outs world, Anthropic, Google, and OpenAI are all willing to sign this commitment, but Meta/X/China aren't. If this happens, the good coalition has a ~six-month lead over the bad coalition. Plan C tells us to have them join together and mutually burn their lead on alignment.
All three companies have a self-image as the sort of people who would do this if it came to that, but in reality they are likely to get distracted or get into fights or delay until it's too late. We should be focusing on changing that.
In one sense, all of this is strictly easier than getting companies to quit unilaterally. It fits with the role these companies like to imagine themselves having. Everyone who previously hoped to win can at least hope to tie for a win. There is low downside risk of accidentally getting superintelligence built by someone who fails to even try our best alignment techniques, or handing over the future to bad actors.
The only sense in which it's harder is that it's a worse fit for (a very pessimistic view of) mass politics, where voters are too simple to consider any worldview more subtle than AI bad = AI companies bad = we should protest AI companies until they stop. I admit that "make AI companies prepare for Plan C" sounds worse at a protest than "demand that AI companies stop unilaterally".
If you believe that mass politics is unable to promote any ideas except maximally-simple, potentially-counterproductive ones, I think this is an argument against hinging a strategy on mass politics.
This post isn't affiliated with AIFP and I think they disagree with some of it.
Plan C is for the leading company to burn its lead. Plan C+ is similar, but all the companies in the leading country work together (maybe because the government is forcing them). Plan B is similar, but the leading country is also sabotaging trailing countries. In Plan D, there is no leading company, and companies can only do alignment work sporadically when they get the chance.
If alignment-pilled companies aren't in the lead, then we end up with the worst version of Plan D, where we have to hope that incidental alignment work done along the way by commercially-focused companies proves to be enough
The problem with China getting superintelligence first isn't just that they might misuse it. It's that, on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
I don't want to get in a fight over whether more good or bad alignment surprises have happened recently, but I also updated on https://www.astralcodexten.com/p/mysteries-of-ai-generalization and https://x.com/_jake_ward/status/2112264244767211854 - not as solutions in and of themselves, but as evidence that good things can happen that I wasn't previously considering. Nick Cammarata's Neural Networks Are Already Clean would also be in this category if I believed it, but I'm currently confused by the debate around it and unsure.
To be clear, this is just the people who have formally considered this question qua forecasters and written down exact numbers. I'm not denying that there are many other smart people, for example at MIRI, who have thought about this hard and believe the opposite.
Again, despite leaning heavily on AIFP terminology and sources in this essay, I'm not claiming that AIFP or its members agree with my thesis in this post, and Daniel Kokotajlo in particular has said pretty clearly that he hopes people unilaterally quit AI companies.
I discuss this further at What Deontological Bars?
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
I'm working on a grantmaking AI that can actually compare the values of different outcomes. It's still in alpha, and to some degree this is a circular argument because I'm trying to get it to mimic/predict my own opinions, but very loosely, it thinks that increasing the alignment-pilled-ness of the leading company from OpenAI-level to Anthropic-level would be worth about 250x as much as the best (low-enough-secrecy-to-mention-here) actual grant proposal it's ever evaluated, which was a proposed $2 million to the AI Futures Project. I think this confirms my "better use of our time than any actual plan we can carry out in those six months" claim.
When I'm modeling this, including on the AI grantmaker, I'm imagining a scale from 1 - 100, where NVIDIA is 10, Meta or X is about 25, OpenAI is about 50, GDM is about 60, and Anthropic is about 80. I imagine a successful campaign to get alignment-pilled people to quit taking Anthropic out entirely, and reducing GDM/OAI to about 40 as they lose their alignment-pilled employees and executives.
I remember how much of a slog the discourse was back in 2015, when the standard counterargument was "you're just a bunch of cultists, no real AI experts believe in this doom stuff". The new standard counterargument of "Of course all the AI experts believe in this, they're just hyping their business for the IPO" is actually a vast improvement!
I asked a supporter how they saw this working - they said that if the US paused, and China didn't seem on track to follow suit, maybe the US would start up again after a few months. But A) this would burn somewhere between most and all of the lead, B) this was just their speculation, and there's nothing in the bill which suggests that the US would do this.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Taking this analogy overly seriously: the biggest difference between Frodo and eg Denethor is that Frodo wanted to destroy the Ring in the end. I'm not sure what the AI equivalent would be - all companies claim they want to use AI to empower ordinary people and achieve broad-based prosperity. Taking the analogy very seriously, Frodo failed to destroy the Ring at the final moment, and was saved only by divine intervention. I don't know what is the AI equivalent of Gollum.
One reason all of this confuses me is that rationalists had an excellent model of race dynamics with China as early as the mid-2010s, when we understood they would be an inevitable part of any future AI development. I distinctly remember going into some old rationalist coworking space sometime around 2020 and seeing "RACE WITH CHINA" underlined on a whiteboard. These discussions included plenty of talk about whether we would be able to coordinate a global pause, but I don't remember anyone ever seriously saying that the problem could be solved if the good alignment-pilled companies or countries dropped out unilaterally.
One of the many ways I have trouble modeling why Samuel would write this article is that it seems like she trusts everyone to drop out of their own accord, except for Anthropic. China? Sure. OpenAI, sure. Meta, definitely. Every single group sponsoring open-weights models in the world at the same time? Yeah, I guess, surely they wouldn't want harms. But Anthropic? We must all rise up to make them do this thing which they would otherwise continue.