I basically disagree with this post.
But do you still stand by the numbers in the post quoted under your name that we have 20% chance of making it in Plan C, and 40% in C+, and only 10% in Plan D? Can you say how big you think the difference is between Anthropic vs OpenAI being in the lead, on the spectrum from C to D?
I always found it surprising that your probabilities seem to indicate that you think that putting in a few months worth of safety effort at the right time is hugely valuable, but you are still opposed to arguments on how it's very important for supposedly more responsible companies to be ahead.
I always found it surprising that your probabilities seem to indicate that you think that putting in a few months worth of safety effort at the right time is hugely valuable, but you are still opposed to arguments on how it's very important for supposedly more responsible companies to be ahead.
I don’t think it’s useful to reason about this unless you’re also decomposing the problem into:
Assuming they do nominally say they’re “burning the lead”, what’s the liklihood this is done effectively:
Imagine you have a company that internally has many employees which are advocating... (read more)
Ask yourself questions like the following:
I think you should add to this list: how would you would feel about putting your name to a half baked argument that's broadly AI pessimistic vs one that's broadly AI optimistic that ended up being widely read? Or perhaps it would be better to be more specific: broadly pro-recusing yourself from the AI industry vs anti, pro-pause regulation vs anti, pro-anthropic vs anti, alignment is probably solvable via a reasonable ordinary R&D effort vs no.
I think one of the main ways these pressures act is via what they make easy to say and what they make hard to say.
I think the positive effects of a single frontier US company stopping unilaterally are larger than you or the AI forecaster think.
Agreed. Anthropic's Pause is the Most Expensive Alarm in Corporate History [Fiction]
[For the purposes of this comment, I will suspend a bunch of major disagreements I have about your background assumptions or explicitly argued things here, e.g. about feasibility of alignment, relevance of empirical alignment, ethicality of doing any AI R&D, etc., and instead try to make a somewhat internal critique.]
It seems like this post argues about a dichotomy "either SURA, or race & align". E.g. your counterfactual is "Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause". But this dichotomy doesn't make sense to me even under your assumptions, as an answer to "What should Anthropic do?". For example, here's a different thing that Anthropic could do:
I mostly agree with you - I don't think my claim hinges on whether current companies are perfect or even necessarily good, it's that part of our goal should push them towards being better (while still "in the race"). Just as we acknowledge that the US government is currently pursuing disastrous policies, but we can think of it as a potential future ally we can win over through activism, so I think we should be doing the same with AI companies (and not just Anthropic - I think that we should be ambitious and try to redeem as many as possible).
I'm not sure I understand your first suggestion - stay on the alignment frontier but don't release. Isn't this equivalent to having powerful internal models but not releasing them (like both OpenAI and Anthropic are doing now to some degree). I worry this could be the worst of both worlds - when the internal models get superintelligent, they can still figure out a way out of the "box", but the world is unaware of them and won't "wake up" in a way that boosts political activism. I don't think having the models be released rather than unreleased boosts their danger much.
I'm unclear on what Anthropic's opinion on / level of work towards a pause is... (read more)
I don't think having the models be released rather than unreleased boosts their danger much.
It doesn't much increase the danger from deployment, but it pushes the race forward. It's like the difference between the policies "You have 100 nukes? Ok, now I have 101 nukes." vs. "You have 100 nukes? Ok, now I have 99 nukes.".
they understand the importance of a pause and think they're doing good behind-the-scenes work, which they won't tell us about
If indeed they are doing the full-hearted version of this, that would change my mind. I simply don't believe it, and the public evidence seems mostly (but not entirely) against this.
I think regardless of whether we trust them, our role is the same - a combination of pressuring every company (including Anthropic) to do this, and building things on the government/mass activism side to get recalcitrant companies in line [...].
I agree we should do this, but not that our role is the same. If core-Anthropic is basically not good--they basically will continue actively trying to do more AI R&D, even as a pause becomes significantly more plausible--then a new-Anthropic person putting efforts toward building them up now is creating the po... (read more)
I'm unclear on what Anthropic's opinion on / level of work towards a pause is. The Pacing The Frontier letter seemed like a statement that they were aiming for it, and the fact that all other major companies immediately voiced agreement suggested that they had held off on releasing it until they'd done a lot of politics behind the scenes and gotten everyone else to agree. My impression of Anthropic, from their public information and the exact phrasing of the cryptic refusals to explain which I get from people who know more, is that they understand the importance of a pause and think they're doing good behind-the-scenes work, which they won't tell us about.
This strikes me an extremely charitable good-faith view of Anthropic, based on their actual record of proposed and supported legislation, why would this be your assumption? Why are you crediting Anthropic with organizing Pacing The Frontier?
Additionally, Pacing the Frontier itself (at least at the corporate/executive level - it seems to have originated among lab employees who genuinely were getting freaked out) can/should be viewed cynically to an extent. The executives from a number of AI companies endorsed it and the companies released various statements, but have they really done anything on that front that they wouldn't have done otherwise? Or was it just a way to take the heat off their backs?
It seems like most of your recommendations are actually very similar to Scott's recommendations.
The remaining point of disagreement seems to be that you don't like the fact that Scott describes Anthropic as the good guys and you point out many ways in which they could be much better.
But I'm pretty sure that Scott is using the phrase comparatively.
He's saying that Anthropic are better than the other AI companies. Faced with a bad bunch, they are the good guys. I don't even think you disagree with this.
I don't think people should interpret your comment as rebutting the core argument that Scott is making in any way.
Stay at the frontier in order to do frontier-level alignment research, but don't release leading models.
Why not release models conditional on staying at the frontier, since most of the risk seems like it exists even with an internal deployment and no release? Because releasing models causes diffusion of ideas to other labs (e.g., o1 or DeepSeek R1) or allows other companies to use their models to do capabilities research? I feel like releasing frontier models has some other benefits (e.g., the ability to uncover and make public evidence of misalignment, the ability for external parties to assess models, the ability for external alignment researchers to use models for uplift on their alignment research), and it’s unclear to me how these net out.
If you're building a monster, let people see it.
Not releasing internal models would mean failing to expose the public to reality and forcing them to think more and more seriously about the consequences of AI progress.
This is not to deny the downside of heating up the race and allowing distillation and spread of ideas. But distillation itself doesn't really shrink the lead, it just seems to.
Seeing things and their consequences with your own eyes prevents a lot of cope.
What we desperately need is for the world, specifically the US and Chinese governments but also anyone who can shift their worldviews, to get ASI-pilled and coordinate slowing/stopping AI R&D. Seeing consequences of cutting-edge models in the wild instead of just hearing reports of internal models could make a big difference in public opinion and therefore odds of an adequate government response.
In my opinion this could well be a fairly large swing in odds.
People change their minds more when they see evidence than from even the best abstract arguments. Remaining in the realm of arguments is fertile ground for the Motivated reasoning, confirmation bias, and groupthink that plague human epistemics.
This is an example of what I'm talking about when I say you take a certain level of risk-awareness for granted that I just don't think actually exists. There is no need for safetywashing, you can literally just say "we're going to make superintelligent AI through recursive self-improvement, and there is no way this could possibly go wrong :)" and you will get less flak than a lab that mentions safety at all.
There's an extent to which you're right that this phenomenon is real and unfair to Anthropic. But to some degree, I think you're probably confusing debate/engagement with flak. Of course I strongly disagree with labs that just say "there is no way this could possibly go wrong." My worldview is so different from theirs that I don't think there is any hope of persuading them; I think the only way to get them to stop is through force of law. I don't think the same of Anthropic. In that sense, the "flak" is actually praise!
To draw an analogy to the most hot-button topic imaginable: sometimes Israel advocates will say "Why don't you spend the same energy criticizing Hamas as you do us? Why are we being treated as the only ones with agency in this situation?" To some extent they're ri... (read more)
Suppose you tried to reason validly from that premise whose reasoning you have not yet followed, saying that alignment by current institutions seems wack, Amodei has been already-selected to still show no sign of understanding alignment basics from 2010, they have no visible comprehension of the basic reasons why "make the AI do our ASI alignment homework" might be hard, etc. Suppose it is the case that leads cannot be burned on alignment. ASI is death, plain and simple.
What policy then follows according to you?
That's you arguing against the US imposing a unilateral halt on further AGI escalation before it gets to RSI and we die before the end of 2027, because you think China would not be moved to follow.
It's not you saying what you'd do instead. What's your optimal policy, supposing you just take as given that these guys can't do ASI alignment?
Okay. What do you do if you think Anthropic destroys the world in 6 months to 2 years, also alignment is not possible?
I can't speak for Eliezer, but at least to me that section then assumes that the labs are on board with the current approach to alignment not working, or are likely to be convinced of such.
You automatically substituted you changing your mind on this issue with the "alignment-pilled" lab changing their mind on this issue. If you assume that the labs will continue to believe what they currently believe, then I don't think any of the arguments in section II go through (or they at least get a whole lot weaker).
Despite our better judgement I and Scott had some debate on Twitter instead of LW about this post. I wrote many long comments, but this one most stands on its own and captures a bunch of my reaction to this post (though far from all of it). I think it makes sense without context, but you can read the context here if you want.
We are both leaders in the same ecosystem. An ecosystem that is extremely massively leveraged on Anthropic (and OpenAI) and general plans to unilaterally build AI ourselves, and you are making arguments primarily directed at those people, both here on Twitter on LW. We just distributed money in a grant round whose public funders are something like 90% Anthropic equity in their net-worth.
And you are advocating specifically for more people in that community to join the labs, and accelerate both their competitiveness and their leverage over the world in racing this way.
And I just don't believe this isn't driving your opinions here. I've seen these dynamics for more than a decade. When I talk to people outside of this ecosystem about our plans to build the aligned machine god in 6-12 months they of course freak out (conditional on believing the machine god part)... (read more)
And of course, the policy you think you are advocating for is not that everyone should race. Just the people who take the risks seriously should race, and everyone else should stop, and then everything will be great.
Huh, is that what you think Scott is advocating for? Seems to me like it really fails to capture the best version of the view.
The hope isn't that people who don't take the risks seriously will slow down. The point is that those people have no reason to slow down (absent some outside force like regulation or geopolitical pressure, which we should also be working on producing, but that might fail) so that the choice is between "everyone races" and "only people unconcerned about the risks race".
Ie if someone who's concerned about the risks chooses to not race, because they are so concerned about the risks, then that doesn't generalize to everyone else stopping, because people who are relatively unconcerned about the risks don't have any corresponding reason to stop.
(What about if everyone was very concerned about the risks? Then we should all coordinate a slow-down.)
Like, see the argument that Holden lays out here, at "the AGI race isn't a coordination failure': https://8... (read more)
Huh, is that what you think Scott is advocating for? Seems to me like it really fails to capture the best version of the view.
Sorry, I agree I am not being maximally clear here. The thing Scott is advocating for is that the "alignment-pilled people according to Scott" get a lead over the "non-alignment pilled people"/"bad guys", so that they can then burn their lead. This means the thing he is advocating for requires the "alignment-pilled" people to race harder than the bad guys, or to somehow slow down the bad guys. In-practice this means Scott is arguing that marginally people should join Anthropic and help them get a bigger lead over the bad guys[1].
There needs to be some mechanism by which the "alignment-pilled people according to Scott" end up pulling ahead for this to be good (by Scott's lights). This is what I am talking about here. Somehow in order for "good" outcomes to happen "the people with the good opinions according to Scott" must pull ahead, and other people must fall behind.
Of course, if everyone acts on this policy, everyone just races as hard as possible because they believe themselves to be the ones with the good opinions. And Scott has identified some coalition ... (read more)
That's one of the things I am saying in this comment, but overall the biggest issue with this whole discussion is the question of "no, who is actually at the frontier?" and "who is 'you'?".
Scott isn't going to be at the frontier. I am not going to be at the frontier. You are not going to be at the frontier. The best you can maybe get are Anthropic executives bound by a giant pile of commitments they made to be very hesitant to stop racing because most of their backers will only benefit from backing them if they keep racing.
In order to do a cost-benefit here it's really important to disentangle what we mean by "you" in "if you race". The thing that is racing is heavily shaped by all kinds of complicated agreements, and your (Lukas Finnveden's) relationship to the thing that is racing will be indirect and confusing and certainly not one of shared identity.
And then the second biggest question, which is highly related to the biggest question is "how will the thing that is racing know that it is at the brink of superintelligence and know when it's time to stop? How will its attempts to stop change what it is in a way that might cause it to continue?".
And then, the questions you raise... (read more)
Right, so reformulating everything from the perspective of someone who isn't the racing entity itself, it's like:
So then I think most of the choice here comes down to a cost/benefit where the main questions are (i) how likely is it that you could build those norms/laws, (ii) how much easier does it become to build the norms/laws if you unilaterally participate in the norms you want to impose vs. if you [help a specific entity] race, and (iii) on the other hand, how much is there to gain from [that specific entity] being at the frontier with a small-to-medium lead in the worlds where the norm/laws never get off the ground. [Where the most important questions feeding into (iii) is what is up with this entity and is it realistically likely to make good decisions.]
Hm, it's interesting that on this formulation, you might well be in a situation where:
Some other exchanges I had with Scott that seem worth copying over for posterity:
Me:
I think you should obviously be willing to risk war if a foreign lab is working towards building ASI. I don’t think you should unilaterally launch nukes without going through a long process of escalation, and only after conventional methods have been exhausted (roughly the same way you would treat any other extremely large national security threat).
Scott:
I think these considerations apply in a small fraction of worlds where the US government is very alignment-pilled but China isn't.
These aren't the ones I expect or think about most, and I haven't thought too hard about what levels of escalation are more dangerous than just trying to pull off a really good Plan C+, which ought to be possible in these worlds.
I think the considerations in my post apply for all worlds outside this small fraction, and that even in this fraction, while I agree with some threats, I weakly lean against threatening nukes, since under my model a really good Plan C+ has an 90%+ chance of working, and nuclear brinksmanship probably has an associated p(doom) > 10% (even granting that most threats will work without having... (read more)
I usually sympathize more with Scott than Oliver in these types of exchanges, but I think Oliver is clearly more reasonable in this one.
if the US were entirely united around solving the alignment problem and actually smart about it (which I think is a precondition in any world where we're willing to threaten WWIII over AI worries)
seems like a load-bearing part of Scott's argument, but of course a country can be united around preventing AI threat (either due to misalignment worries, or due to worries that no human, especially no other enemy human should get the power) without being "actually smart about it". Being freaked-out and actually smart about the thing seems much harder than just being freaked out, so there can be a big chunk of worlds where the government is freaked-out enough to threaten war, but not smart enough that we can trust them to channel their freaked-outness into burning their lead on an actually reasonable alignment plan.
Another exchange:
Lance Spencer:
Do you believe Scott himself was on the "let's not get involved with DC" side of the fence?
Richard Ngo has a pending essay on why he thinks getting the government involved is a bad strategy (I think), but I don't know the nuances.
To me, either the pause/pace or the alignment strategies almost certainly need a DC angle. (Unless it turns out somehow alignment was actually much easier than it seems to be at present.)
Me:
My model of Scott thinks that the existing strategy in DC of lots of people trying to play various inside games is great, and he is supportive of the people who have been trying to get the people who are louder about the concerns to shut up, because he does think that being more open that way will cost lots of social capital.
I think everyone agrees there should be a DC angle, the question is how much does that involve people seeking power and doing various inside-game like things, and to what degree does that involve people speaking openly about the risks and building popular support. These two have historically been very drastically in conflict (and the former coalition has been enormously dominant in DC EA as far as I can tell).
Scott:... (read more)
Crossposting a long comment I left over on Against strawmanning unilateral pauses, just to have my comments in a consistent place:
Scott:
[...] Just to make sure I'm not accidentally mischaracterizing you, do you agree with Geoffrey's position above (I'm nervous summarizing, and you should double-check to make sure I'm getting it right, but I'm understanding it as saying that the moral thing for companies to do would be to trial balloon a brief pause, maybe a couple of months, to see if other companies join in, but if not, they should continue racing)? Or do you think companies which are trying to be moral should stop racing permanently in a non-trial balloon way?
Me:
My position (condensed into a few paragraphs, though of course I believe much more on this topic):
The moral thing to do for a company is to spend a large fraction of their resources on slowing down everyone equally, while also shouting desperately from the rooftops about the risks and the need for a pause (substantially above the minimum floor of what Anthropic did in the DoD dispute, and what Uber did in various municipalities to use its platform to lobby for laws that enable ride-sharing, which are inadequate but rea... (read more)
Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.
Huh. It turns out that the ACX Meetup Czar thinks everyone is underestimating ACX meetups, up to and including Scott Alexander. Outside view that's probably a sign I need a psychiatrist or something.
The community was growing steadily right up until Covid-19, had a dip just like every other social community, and then started growing again. The long slow march through institutions is ongoing and pretty clearly working. We're a a movement that primarily recruits intellectuals in the middle of their undergraduate degrees. If you're in undergrad in 2010 you will finish your PhD in 2020 and be vying for professorships in 2030. The rationalists skipped ahead a bit (it's not like we weren't attracting PhD students and professors in 2010) but if we'd managed to hold the hockey stick off until 2040 we were going to be everywhere in the university system. And that's just the colleges!
If you were fifteen in 2015 when HPMOR finishes then you're in your mid-twenties now, which is why ... (read more)
This post makes plenty of good points, but I think your argument against racing to superintelligence being deontologically evil is weak. Roughly speaking, you draw an analogy between developing AI and using AI as a consumer. There's a few things I could say about that analogy directly:
-The first is that the analogy has some merit, and that I don't really feel amazing about my own heavy AI use. And I have at least one friend who has never used an LLM, explicitly because he doesn't want to contribute in any way to the race to superintelligence and human extinction. Maybe some people would call his choice foolish, but I have a lot of respect for it.
-The second thing is that there's billions of users of AI, but only thousands/maybe tens of thousands of frontier AI developers. It makes a lot more sense to try to coordinate some kind of prohibition at the latter level and than at the former. And independent of the mechanics of a boycott, I think everyone has a reasonable understanding that the developers of AI bear a lot more individual responsibility for outcomes around AI than consumers do.
But mostly, I think this deontological prohibition thing deserves more of your attention than you... (read more)
The most common sentiment I hear, once I've fully explained the situation and insisted that no, it's not them hyping, it's not about the IPO, this is something they really truly believe, is "those people should be arrested."
Notice that this worldview ends up cashing out to "if there is a technology that is potentially existentially dangerous, then only people who are unaware of the danger should build it - anyone who is aware of the danger must be stopped."
It is not enough to "have sympathy" for Anthropic here, the 'normie position' you have described here is straightforwardly suicidal. I know everyone's happy when they get to be on the side of the common person, but sometimes the common person's first take within 0.5s of hearing of an issue that they don't really care about, is just very terribly and dangerously wrong.
Edit: to try to spell this out a bit more, the reason I think the worldview cashes out in the way I described is because the people in question are AI-risk-skeptical and therefore are not supporting government action to stop all AI development; and yet they are supporting arresting just the people who believe it is dangerous. This is what makes this 'normie' worldview uniquely worse than either SURA or, uh, anti-SURA / alignment entryism.
It's basically Copenhagen interpretation of ethics applied to AI safety
Surely normies would want to stop everyone from building existentially dangerous technology. That doesn't conflict with the belief that people who know the technology is dangerous should be punished more harshly.
If a sane adult shoots somebody with a gun, they go to prison, but a small child who does the same thing doesn't, because they don't know any better. But of course it's important to prevent innocent people from being shot, whether the shooter is an adult or a child.
Extending the analogy, adults sometimes shoot people in self-defense. Sometimes they shoot recklessly and accidentally kill an innocent bystander. Whether/when any given incident should be considered grounds for arrest isn't clear-cut. But as a first pass, it isn't stupid to say that "people who knowingly take actions that will likely cause others to die should be arrested."
sometimes the common person's first take within 0.5s of hearing of an issue that they don't really care about, is just very terribly and dangerously wrong
I agree. The common person loves rent control; that doesn't mean I love rent control. I'm not bringing up the normie position in order to argue that their position is right; I'm bringing it up to point at the second-order issues that are going to arise as a result. If, once everyone finally wraps their head around what you're doing and why, most of them are going to want to see you arrested, then that's... a problem. It's going to cause issues for you. And it's going to cause issues for the rest of us, too.
That's why I think it might make sense for Anthropic (and anyone else who shares the safetyist worldview) to unilaterally stop. I understand the game theory looks bad. But if Anthropic stopped, their position and motivations would suddenly look coherent to the everyone else, when right now it doesn't.
the 'normie position' you have described here is straightforwardly suicidal
Is their position straightforwardly suicidal? Or is Anthropic continuing to race towards superintelligence, when the arguments in list of lethalities + what... (read more)
That's why I think it might make sense for Anthropic (and anyone else who shares the safetyist worldview) to unilaterally stop. I understand the game theory looks bad. But if Anthropic stopped, their position and motivations would suddenly look coherent to the everyone else, when right now it doesn't.
Why not have all the people who didn't join Anthropic for principled reasons go ahead and do the persuasion to the normies, since their position is so coherent, and the normies will recognize this coherence and virtuous deontology and therefore see the light?
As someone who believes that alignment success is highly responsive to how much care goes into alignment do you see why it is insanely frustrating for me to hear people say "hey, let's just have everyone who cares about alignment quit, maybe that will make the normies notice"?
Is their position straightforwardly suicidal? Or is Anthropic continuing to race towards superintelligence, when the arguments in list of lethalities + what we've seen in 2026 of RLVR'd AIs + the total absence of a theory on how to preserve what little alignment we do have through RSI + even a tiny dose of Murphy's law, the straightforwardly suicidal position... (read more)
Why not have all the people who didn't join Anthropic for principled reasons go ahead and do the persuasion to the normies, since their position is so coherent, and the normies will recognize this coherence and virtuous deontology and therefore see the light?
I don't completely understand, sorry. What would they be persuading the normies of? That superintelligent AI is very dangerous? That building it would be wrong? They're on it, and also, the normies are mostly persuaded. (See question 54 here, "AI will destroy humanity." That survey is from 2025!).
The list of lethalities just doesn't justify such an extreme level of pessimism. It lists several potential problems, but "you train the AI on a bunch of examples of it being nice, and when it crosses over to the new superintelligent domain, it luckily keeps on being nice" is still an entirely plausible AI success story that you can be made more or less likely depending on how much care goes into its training.
Maybe we'll get lucky, I guess. But there's a lot of reason to think that the outcomes we want are a small target, that the things we do now won't generalize to superintelligence, that empiricism doesn't work very well in an adver... (read more)
The problem with consequentialism as a moral system is that it's not a moral system: It's a system for deciding your own actions. A moral system is a system for determining which other people to socially reward and which to punish/ostracize. There are two ways to turn consequentialism into a system that does this: You can base your judgements of people on their beliefs about the consequences of their actions, or on the actual consequences of their actions. The problem with the former is as you describe; people can always justify their behavior on the basis of some belief that it will have positive consequences, and you have no way to reliably know whether they're lying (whether consciously or subconsciously). The problem with the latter is that people get randomly rewarded/punished for far-flung consequences of their actions they could not have foreseen, making the system less reliable overall and decreasing the incentive to actually try to do the right thing.
The way morality works in real life is that we generally only judge people based on common knowledge consequences of their actions; consequences that everyone roughly agrees on and nobody can deny knowledge of. For example, if... (read more)
but one thing I want to focus on in particular is that deontology is "normie ethics," and when you're considering doing something that will affect everyone, including every normie, it's important to take into account how they will see it.
I think implicit in Scott's argument is a question of consistency.
This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".
I can definitely believe that when you talk to people about these types of things they react by saying "those people should be arrested." I've also had multiple people, who probably would count as "normies", tell me that once someone has a net-worth of over 1 billion dollars, they should immediately be executed. I also know that things which I don't consider to involve hurting others are illegal or otherwise considered morally reprehensible (drug use, pay-for-sex arrangements, etc.). Many people would probably say donating to something l... (read more)
The analogy I'd choose is driving. It's hard to define exactly what it means to be a good or bad driver, but at least as important as raw skill is predictability, by which I mean how predictable you are to everyone else. If you drive 30 above the speed limit on the highway, weaving between lanes to pass the other cars, is that fine so long as you are skilled enough to avoid an accident?
I like this analogy, but one could argue that the current global AI situation is more like a New Jersey highway: everyone is speeding and being aggressive. If you don't do that, you're the unpredictable one, and you might get rear-ended and cause a pileup.
If we extend that analogy, Anthropic is speeding and being aggressive like everyone else, but also advocating for "hey we should set a new speed limit, one that's maybe higher than the road-death-minimizing ideal, but which can actually work as a Schelling point and be consistently enforced". Put that way, it's not "weird driver" behavior, it's a totally reasonable stance a lot of people have about US highway speed limits.
On a different note, IMO you're mostly making a consequentialist argument for deontological decision-making. ("Anthropic's strate... (read more)
Similar to my comment criticizing Dean Ball for his self-positioning on AI safety discourse censorship as a nobody too easily swayed by prevailing currents, my biggest problem with this line of reasoning is that it makes it sound like Anthropic is a small player among many in a crowded race, when it's more like they're one of the biggest proponents and participants of the race (including before Anthropic was founded, at OpenAI). Naively, I think your intuitions for what to do if you're a small fish in a big pond should be pretty different if you're like the biggest fish in a smallish pond. Like there should be more personal responsibility for the collective situation, and also more suspicion of these "big-world" intuitions being appropriate here.
(One obvious disanalogy between actual racing on highways and metaphorical racing is that in a literal race, being obviously ahead in the race and actually faster than everybody else isn't obviously more dangerous, and in some ways actually way safer)
I am glad you wrote this post. I expect the post itself and the ensuing discussion will be good for the field, and I share some of the concerns you have about epistemics in AI safety/policy discourse. Here are some places where I disagree or have comments:
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking.
Interesting. It feels like just a couple years ago the common belief was that pretrained models were certainly not aligned and SFT/RLHF post-training was needed to shape them into good assistants. But this post-training certainly wasn't perfect, given things like jailbreaks and sycophancy. I don't think we ever got the "actually good, HHH chatbot" before companies started doing RLVR and therefore introducing a bunch of reward hacking problems. If I were you, I wouldn't put much epistemic weight on this model.
Throughout the post you discuss how going all in on SURA makes Plan C much more difficult. Unfortunately, I think the effect is also real in the opposite direction. Continued capabilities escalation and cap... (read more)
Thanks for these thoughtful criticisms. My responses:
Interesting. It feels like just a couple years ago the common belief was that pretrained models were certainly not aligned and SFT/RLHF post-training was needed to shape them into good assistants. But this post-training certainly wasn't perfect, given things like jailbreaks and sycophancy.
I may be bracketing this wrong. I mean pretraining and things like SFT/RLHF which draw out different versions of the pretrained persona. I think there's been decent progress on jailbreaks and sycophancy since then, and they don't seem like the kinds of problems that affect potentially-dangerous superintelligences. AIs are godshatter in the same way as humans, but in order to be dangerous in a scheming way they have to have the human ability to build themselves a self out of the godshatter and point it in some direction. I'm concerned about things like RLVR that mean the "self" they make is one with anti-human goals, but not as concerned about things like jailbreaks - I don't have *proof* that jailbreaks would be ego dystonic to a superintelligence, but that would be my guess.
Throughout the post you discuss how going all in on SURA makes Plan C ... (read more)
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
This is exactly what happened with biological weapons. The US decided to unilaterally disarm in 1969, and then pushed for an international agreement in 1972.
It's also what happened with CFCs. The US unilaterally decided to stop using them in 1978, and then the international agreement came in 1987.
Unilateral disarmament is a costly signal that you actually think that this is too dangerous for anyone to use, and establishes you as a moral authority on the issue internationally. This is a viable strategy to get an international agreement.
Unilaterally banning biological weapons made more sense because they're terrible weapons, e.g. too easy to cause 'friendly fire'.
This doesn't seem like it would have worked for nuclear weapons, or almost any other kind of weapon historically.
I'm not sure unilaterally banning (the race to) ASI or RSI couldn't work, but the tradeoff is very different.
Let's check the history of the US chemical weapons program on Wikipedia too. (I'm less familiar with this history, and am not doing a careful check.)
The US unilaterally announced a no first use policy in 1969. The US unilaterally committed to destroying its chemical weapons stockpile in 1991. The Chemical Weapons Convention was signed in 1993.
Norway unilaterally banned cluster munitions and later became a key leader in the eventual international convention against them.
That was just the first conventional weapon that has some international ban I thought of. I've now also thought of anti-personnel landmines. Looks like that's more complicated - there were unilateral disarmaments but they were more closely entangled with treaty process - e.g. Austria unilaterally banned them shortly before the international treaty, but was already a leader in the treaty drafting process at the time.
Except maybe Zimbabwe; not sure about why they unilaterally disarmed.
Maybe a more "selfless" unilateral move would be the US, Russia or France banning the export of landmines (harming their own arms industries for humanitarian reasons).
While unilateral disarming biological weapons helped a treaty be signed with wide international support, that treaty completely lacks any meaningful enforcement mechanism.
As a result, the Soviet Union never disbanded their bioweapons program, and by all indications (according to the US State Department, most recently in a 2024 report) that bioweapons program still continues to this day within Russia.
*I actually think the unilateral disarmament of bioweapons is an example that favors Scott's argument.*
I'm a proponent of SURA, and didn't find much to update on in this post.
According to me, the core issues are 1) whether or not the company alignment plans will succeed and 2) how close we are to the point of no return. On the first point, I think it is very low, for detailed reasons, and I think somewhat more importantly even if you take the company self-reports at face value it is substantially lower than is legitimate to accept. You, as a patient considering a cure to aging, might accept a 10% chance of death in exchange for immortality, but making that trade for everyone at once is the sort of thing that probably ought to be put to a vote. (And an independent panel of experts might think there's a substantially higher than 10% chance of failure.)
I also think we are uncomfortably close to some points of no return. Current attempts at RSI only deliver modest speedup; if there is a 'sharp left turn', we haven't hit it yet. But even if we don't have lightning-fast takeoff, many people at the labs suspect that the models will be able to replace them at some point in the next year, at which point employees lose all of them leverage on the labs. Once there are models capable of autonom... (read more)
I do realize that the race dynamics are relevant, and I still think it makes sense to push for SURA.
Some headlines of my thoughts on China:
The premise here is that Anthropic does not have a realistic ability to reject that gamble.
Oh? What have they tried? Arguing that the federal government shouldn't be able to stop them?
[Edit] On reflection I think this comment was meaner than I'd like it to be. The core frustration behind writing the initial version of it was that I don't think Anthropic looks like an organization that attempted to reject that gamble and failed, and instead looks like an organization that wants to pursue the gamble. I suspect this is important for modeling Anthropic. I think people who work at Anthropic because "well obviously regulation won't happen" and "well obviously international coordination won't happen" should pay close attention to whether Anthropic is pushing for or against those things.
with no plan for what happens if race dynamics continue anyway.
What? The plan is "stop them". Israel doesn't meekly accept the existence of the Iranian nuclear program; it and its allies use a variety of means to dissuade, deter, disrupt, and destroy that program, on the view that it is a threat to their continued existence. I don't think it makes sense for individuals to act on their own, here--Ted Kaczynski did not succeed and should not have tried. I think that it is basically possible for governments to declare war on rogue AI systems and their creators.
I don't understand where we disagree. I agree that your plan is to stop them, but presumably your odds of succeeding are less than 100%. This leaves open the possibility of a plan for what happens if you fail, which I'm complaining you don't have, and proposing an example of.
In the post you repeatedly mention (paraphrased) "what if we don't get a treaty" and "seems like they are sabotaging any efforts if they don't get their perfect treaty" and generally fully equivocating "plan A" with "get a treaty" and "plan C" with "burn your lead" with nothing in-between.[1]
I think Vaniver is saying "there are many more worlds where you can buy more time and get more traction on the problem than via a treaty, which this post seems to mostly ignore".
See for example this person: https://x.com/ApriiSR/status/2116952850417004564
See also this critique by Boaz: https://x.com/boazbaraktcs/status/2116969685178634347

I don't know, maybe you are talking about someone totally different than me or MIRI or whatever, but the post seems to create a pretty weird dichotomy in strategy-space that I think few people occupy, and that's what Vaniver is saying.
Some specific quotes:
If we demand that the US cease AI research unilaterally (ie without a mutual pause treaty with China) [...]
All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that th... (read more)
Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies.
What do you mean "more complex"? Coxon's resignation feels like it directly demolishes your argument in this section. You might remember Kabir Kumar writing Please Do Not Resign In Protest the day before Coxon resigned. I think Evan's tweet amplified it--but I think if Evan had also resigned, that might have done even more.[1]
But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight.
Neither Eliezer nor Nate worked at AI companies, and nevertheless they get media appearances.
Evan is one of a few Anthropic employees that I wouldn't suggest quit, mostly because I think Anthropic's overall alignment plan is doomed and Evan is one of the few people trying to accrue evidence for this point. [Edit: this might have been a stale impression? I remember thinking this was his main focus a few years back, but a friend points out that he seems on-board with the automat... (read more)
I think it's clearly more complex. On the one hand, it's good for people to sometimes loudly quit, to raise awareness. On the other hand, if everyone quit now, then we would have one big media cycle, and if that doesn't solve everything, there are no more shots at loud quitting. If people never joined the companies, as I feel SURA advocates often advocate, then we wouldn't have had any media cycle about quitting.
To be clear, it's still very reasonable to argue that on the margin, more people should quit right now, and fewer people should join/should have joined in the first place. I think Scott is arguing against a stronger, deontological statement than the argument on the margin.
(And yeah, I agree the Kabir Kumar post was bad, I think that was knowable at the time too.)
On the other hand, if everyone quit now, then we would have one big media cycle, and if that doesn't solve everything, there are no more shots at loud quitting.
But by your hypothesis, loud quitting isn't effective. In that case, what's the value of saving that tool for later?
If people never joined the companies, as I feel SURA advocates often advocate, then we wouldn't have had any media cycle about quitting.
Sure, but if people never joined the companies, there wouldn't be companies!
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
Sounds made up.
If there was any particular wisdom that was most-conventional...
( And not a plan involving human brain uploading or human intelligence augmentation, sitting on a different axis entirely than the A-C axis of this post.
And not Bostrom writing about what it would look like to take the Vulnerable World Hypothesis seriously, sketching a world much stricter than your Plan A.
And not the dreams of AGI projects of varying cred who had their own plans for how they'd do it right themselves. )
it was that we would Figure Out Alignment, from the mathematical underpinnings to the philosophical questions to the plausible implementations, in advance of any particularly clever AI. We would solve the problems and the solutions to those problems would imply the structure of the intelligence that would fill the galaxies with flourishing civilization.
That was the most-conventional wisdom, as I recall it. The intervening years have killed it off as surely as the dodo. And though you'd be right that it doesn't much sound like "... (read more)
But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.
Tolkien (through Faramir) explicitly addressed this:
‘But fear no more! I would not take this thing, if it lay by the highway. Not were Minas Tirith falling in ruin and I alone could save her, so, using the weapon of the Dark Lord for her good and my glory. No, I do not wish for such triumphs, Frodo son of Drogo.’
‘Neither did the Council,’ said Frodo. ‘Nor do I. I would have nothing to do with such matters.’
[p. 695 of this pdf]
The rest of this section does not describe Frodo Baggins. There's a missing mood, and it's the entire character.
Frodo does not claim that you can trust him with ring. He attempts to give it away, to Gandalf [p. 85], to Galadriel [p. 389], and to Aragorn [in the movies only]. Each of them refuses. No one can truly be trusted with the ring, and multiple characters wish that it had never been made. Frodo accepts that he has it only because fate brought it to him [p. 80], and then Frodo reluctantly chooses to accept the terrible burden of being the person who attempts to destroy the ring [p. 294].
Frodo never uses the ... (read more)
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
I want to flag that modeling the implications of competition + potential coordination with China on what frontier US companies should do now is an area of ongoing research for us, so I wouldn't state specifics confidently.
That said, it is true that we think it would be good for US companies to pace/pause right now, and poor security is one argument in favor of that. My personal opinion is that it's unclear what level of immediate pacing is best.
One important disagreement with Scott's summary of our position:
Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
I currently think that it's better to pace than to race even once companies have good security (as I mentioned, I'm uncertain about the optimal extent of pacing). There are various benefits to not going all out on the strategy of racing then pausing at the last moment; for example: (a) practicing for later pacing/pausing, (b) we're uncertain about when the point of no return will happen and it seems plausible that it's within months, (c) it's uncertain whether companies will actually pause at what we think is the right moment, (d) it's the more cooperative thing to do w.r.t. China and my current guess is that it increases the likelihood of. a US-China deal (I know you discuss this some in your post, I haven't read it in full carefully yet).
(I didn't mean to imply that I agreed with all of Scott's summary of our position in the above comment, but perhaps it did, sorry. I haven't taken the time here to write up our our full current position in part because it's evolving over time.)
I’m trying to evaluate if I disagree with this post or not. It’s unclear, because I’m not sure whether I agree with SURA. But I do think the labs, and the employees at the labs, are morally obliged to radically change course in one way or another.
Let’s take Anthropic, because they’re in the lead and because they are widely regarded to be the most alignment-pilled company (though they may be very weakly alignment-pilled compared to the actual difficulty of the problem.)
Do I think that Anthropic should pause unilaterally, right now?
I admit that I think it is pretty plausible that they should. But whether I would recommend they pause at this moment does depend on the execution and the specifics of their plan. If there is some internal modeling that leads them to think that they are better off pausing three months from now rather than today, because of the forecasted usefulness of the capabilities they expect to gain in those three months of capability development, then I could certainly imagine being persuaded. (Though importantly, if they were planning to pause in 3 months, I would expect them to be doing a LOT of both internal and external preparation for that radical shift.)
Similar... (read more)
The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.
I've investigated the history of the environmental movement before, and this is not a good description of it.
Environmentalists have always had significant infighting, and some factions of them have always been against something one might call capitalism or progress.
For example, from Wikipedia (before the time period I focused on):
In the first decad... (read more)
My preferred plan is for there to be much more accountability to the public / government(s) w.r.t how frontier AI is developed and used. I don’t know if you include this under SURA or not. The end goal is pretty similar but the route is pretty different (ie we systemically force / incentivise a slowdown rather than relying on companies to do it individually).
One of my biggest concerns with Plan C is that I think it’s very hard to differentiate an “alignment pilled” company that is accruing a lead to burn it, from a straightforwardly power-seeking company that just wants to accrue as much lead as possible. The actions taken are identical; furthermore the latter can always just say they don’t think the time is right to burn the lead yet.
For this reason I think “alignment pilling” the frontier AI companies is grossly insufficient. I do not want to have to trust somebody’s voluntary, non-binding stated intentions when the stakes are the possible extinction of humanity.
You seem to downplay this concern with a fictional example (Frodo Baggins) and a handwavy statement of “lots of people have faced temptations in the past”. I don’t find this very convincing.
The people I know who joined ... (read more)
But it seems like this same argument should apply equally well to the AI companies
i'd have thought the deontological prohibition on building a world-killing tech is much bigger on that on using a pre-cursor to such tech
(Scope status: responding to one point, not necessarily endorsing other points not mentioned)
One thing I dislike about centering the Plan A/S vs B/C+/C (vs. anything else) debate is that it misses the point that a staggering portion of all things someone reading this could do in the next 12 to 18 months are beneficial to virtually all such worlds. I take @Vaniver 's point elsewhere that some of these, like unilateral quick pause to avoid rushing into RSI, are indeed incompatible with B/C+/C.
But even there, almost all the work one would do to prepare for such a pause, to make it more coherent and implementable and thinkable, is also useful in the other worlds as a negotiating chip, as a pacing mechanism/"dimmer switch", and yes, as a risk-mitigator for racing.
I don't object to people having big strategic debates, they're good, they're great. Love 'em.
But I come away from this essay suspecting that the same people you are addressing in your introduction, @Scott Alexander, the people who are still wrapping their hands around this problem, will take the implicit message here of, "I need to focus on deciding what side I am on this debate," instead of just doing good work that's r... (read more)
I don't understand. Isn't Scott mostly advocating for more people to do things like start Anthropic and get more power at the labs? The position of "only do things that are robustly good across both power-seeking strategies at the labs and across awareness-raising strategies in the world" seems very limited.
Like, indeed, Scott seems like he is actively against trying to raise awareness in a way that might risk a domestic pause but not an international pause, and I certainly think your efforts in DC are more likely to just cause a domestic pause than to both cause a domestic and international pause. Are you going around and telling policymakers that they should not slow down unless they can get China to slow down? Are you telling them they should make sure to not mess with Anthropic in-particular because they are the good guys? Are you making sure to treat the researchers, including the capability researchers, at Anthropic with respect and to not imply they are bad people for building ASI and pursuing RSI?
If you aren't, then I think you aren't acting in a way that is robustly good by Scott's lights.
(I am not saying Scott is saying that you shouldn't do the work you are doing, but I don't think your work is "robustly good" by the lights of the strategies that Scott is arguing for here)
I've heard people say that maybe this would shock China so much that they would realize we were honest in our fear of superintelligence, and change their minds. I don't dismiss this possibility, but it also seems possible that China - having seen us place overly burdensome restrictions on nuclear power, infrastructure, and housing because we are a "lawyer state" vetocracy - would not be very shocked to see the US' leading socialist place restrictions on AI too, and would continue pursuing AI for what they think it's good for - a dual-use technology which could give them a major although not world-changing military and economic advantage.
I think this is misrepresenting the strategic situation.
If the US government became existentially afraid of an intelligence explosion leading to a rouge superintelligence, it would not just ban superintelligence domestically and then politely ask China to also stop. It would update the Nuclear Posture Review and credibly signal that if any country in the world is allowing a runaway intelligence explosion within their borders, the US will use military force to shut it down—because we consider a threat tantamount to releasing a bioweapon.
In all likeli... (read more)
In this version of the Plan A/Plan C categories, would it be fair to say that right now almost all AI safety money and resources are going toward neither? Almost all resources are doing toward (1) technical research that does not much help with either plan, or (2) policy work that does not much help with either plan. Or maybe it would be more accurate to say that most AI safety work slightly helps with both plans but is not efficiently directed at either one, e.g. it might make risk more obvious at the margins, which increases the odds that people take the risk seriously before it's too late.
It looks to me like most resources are going toward a world-model where prosaic alignment just works, and there is no need for a globally coordinated pause, nor for the leading companies to pause and burn their lead on more alignment research.
I'm not sure.
I think a lot of politics is vague coalition-building and Overton-Window-shifting, and I think that (for example) having more pro-regulation politicians in power, or more local PauseAI groups, will make it easier to get Plan A (when the window shifts) even if the money isn't earmarked "boost Plan A".
Likewise, some of the political fight is just making sure that people know what's going on (eg frontier company transparency legislation) which could be useful in the Overton Window shift that might one day get us Plan A.
And Plan C+ (get all American labs to cooperate in burning their lead over China) takes a level of coordination between US labs that could only be achieved by government (if only to negate antitrust rules)
I'm less sure about alignment. If you think the future auto-alignment-researchers will be one million times better than humans in every way, there's no point to doing alignment research now. But if you think that:
There's a growing effort toward technical/policy work on "compute verification" in particular, which is necessary for Plan A and possibly Plan C+ (US-wide coordination). Per Scott's post, emphasis mine:
Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty.
"Compute verification" is the technology necessary to track chips used for AI and verify that they're only being used in agreed-upon ways. For example, if there were a training pause, compute verification means that the US could prove to China that all its datacenters are only running AI inference for existing models, not secretly training new ones. All chips produced would be accounted for, and an inspector could walk right up to each of them in a datacenter and verify via some technical means that they're running only inference. (more practically there are a variety of plausible technical means of scaling verification, see this AI Futures Project supplement for more details)
The AI Futures Project themselves are very interested in boosting work in this area and offer to connect those interested in working on it to relevant funders/experts. L... (read more)
Haha I admit to my comment being partly motivated toward just trying to drum up interest in compute verification. Sorry for assuming you didn't know about it.
Re: <1%, I had the impression it's larger than this given how much of a priority it seems to be according to various orgs in alignment. Doing some light checking:
This is a great post. I have long felt that this set of ideas is wrong on many different levels and I think you do a great job of walking through them. I do hope some people reading this who have kind of drifted into a SURA viewpoint through osmosis will have a sudden shock where they say "wait, there were a lot of bs reasons which were feeding into my belief" and update hard away from it.
How do people feel about SURA as an acronym for this set of ideas going forward? I've felt like we need a neutral nametag for this set of ideas and SURA seems as good as any, with the main downside being that it's introduced in an "against" post.
I've also thought of "alignment entryism" vs. "alignment exitism" - entryism meaning you think alignment-pilled people should be at the frontier, exitism meaning you think they should exit.
I'll probably use SURA going forward unless people object.
I object to "SURA".
It's catchy—Scott has a great feel for language, of course—but it also feels implicitly derogatory, and it's not a good sign that by the time I finished reading this post, I couldn't remember what it stood for. ("Stop Unilaterally + Raise Awareness")
Kind of reminds me of other negative-valence, outsider-popularized political acronyms like RINO or TERF or SJW. There are positive-valence political acronyms (YIMBY, a twist on negatively-charged NIMBY), but even such acronyms chosen by the relevant ideological groups themselves (EA, MAGA, BLM, DEI) tend to acquire strong political charge, which means that for many people, they're negative-valence and thought-killing.
I would prefer terminology that can't be so easily decoupled from its intended meaning.
Naming an idea is always a minor act of aggression. It turns it from a subject that's doing the criticizing into an object that can be criticized. And it makes it sound like an Ideology (bad), whereas unnamed ideas might just be grassroots collections of people who happen to have converged on the same thoughts (good).
I tried experimenting with this post without naming the idea, but it was clunkier and relied too heavily on the Plan A vs. Plan C dichotomy, which isn't really right (AIFP uses those terms more specifically than I do, with Plan A being only their preferred version of the pause).
I don't think naming an idea is always aggressive against it. Was naming Effective Altruism aggressive? Arguably to ordinary charity, but not to itself.
Rather, I think naming an idea makes it salient and easier to discuss, if perhaps sometimes harder to reason about. Naming does make ideas easier to demonize, but it also makes them easier to lionize. Perhaps more often it's a criticizer who comes up with a name that sticks ("marxist", "quaker") but that's not always the case ("federalist", "pro-choice" and "pro-life").
What bugs me about SURA is mostly that it doesn't naturally parse into the actual position being criticized, because it's an acronym of a phrase no one used.
I do admit to a lack of an obviously better suggestion, though.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
For what it's worth, I think Daniel Ellsberg makes a pretty compelling argument that it would make sense for America to unilaterally destroy all their land-based ICBMs, but keep a few nuclear submarines to be able to credibly threaten destroying some major cities. This way, Russia would also have less incentive to maintain such a giant nuclear arsenal, as a major justification for having a large arsenal is that it is needed for taking out as many enemy ICBM sites as possible at the beginning of the war, to minimize damage to your side.
This directly happened under the INF and START arms reduction treaties, to the extent that Russians were traveling to American air bases to hand count warheads.
Even this otherwise unprecedented level of compromise, however, has only reduced U.S. and Russian arsenals to… the second and first largest arsenals on the planet. It’s very difficult to pare down forces to the point they open you up to strategic vulnerability (ex: putting all your eggs in one basket with SSBNs). Imagine if the Soviets had precommited to not invest in or scale their ICBMs in the early Cold War—they might well have gotten crushed by a paranoid U.S. committing to a first strike against their obsolete bomber forces.
I think the natural equivalent would just be to threaten to destroy the data-centers of any foreign nation that is building ASI (using conventional weapons and nukes if really desperately necessary for some reason), while stopping any frontier training yourself. The deterrence equilibrium doesn't have to play out only on the AI level.
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
This doesn't match my recollection and seems inaccurate to me. Below I share my detailed thoughts so people can tell me whether they think I'm misremembering things or whether it's Scott's characterization that's inaccurate (or some of both).
(1) My recollection is that for most of the time prior to 2023, most people concerned about AI xrisk weren't even thinking about the possibility of a decades-long global pause. ASI seemed so far off that there was little reason to think pausing would be necessary back then, and certainly didn't seem necessary then.
(2) Most people weren't saying we should be preparing for Plan C, which as Scott says, involves attempting to solve alignment at the last possible moment while burning the lead.
Rather, I think that most people were saying we should just attempt to solve the alignment problem well in advance of humanity's ability to create ASI. For example, Stuart Russell published a book in 2019 arguing that we should stop building AIs that optimize for fixed objectives and instead only b... (read more)
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking...
This whole section seems really confused. Pretrained models are text-completion engines and have no intrinsic desire to "imitate humans" (except insofar as most pre-2023 training data is human-written). RLVR came of age around 2025—famous cases of misalignment like Sydney/Bing, GPT4-o, and Opus 3 (as in your "Claude Fights Back" essays) pre-date this. There's no reason RLVR in particular ... (read more)
People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me.
Wait, but the ai safety community would have been way bigger! And become way more senior in their careers.
And similarly, if we pause for 1 year today, we get big community growth and seniority gains
(I agree time later is better. But time in the 2010s and today is still v good)
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
This seems importantly disanalogous.
The more apt analogy would be...
We want to stop building more nukes, we hope that you'll join us in this.
We're serious about this and want to show that we're serious, so we've stopped manufacturing more nukes. We would like to negotiate with you to do the same. But if those negotiations fall through, we may go back to building nukes. (And by the way, we have WAY more uranim than you do, so if you want to make a competition of it, we have a big advantage. But we think competing on nukes is bad for all of us, and we would prefer not to do it if we can.)
The key difference is that destroying your nukes permanently and ~irreversibly[1] puts you in a substantially worse negotiating position. Holding off from building more nukes, in contrast, is easily reversible, and barely hurts your negotiating position.
Follower-conditional leadership is an underused strategy, but I think should often be pretty close to the default when facing coordination problems. You can take on some cost to you... (read more)
Would they have preferred a world where all of these people quit the day before, and letters like this never got written?
This seems like weird rhetoric?
People who want lab employees to quit want them to do it loudly—raising the alarm about the problem. If all 1386 pacing-letter-signers had quit in protest, stating loudly and publicly that the companies are endangering our lives, I think this clearly would have had a much larger effect than the (meaningfully large) effect of the pacing letter. It would be a way stronger signal, since the people involved would be making a more substantive sacrifice.
Perhaps the tradeoff is not worth it because, not having quit, those employees can still take actions within the company, and can still quit in protest later.
But to the extent that the pacing letter was good, quitting in protests would be better.
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
This seems kind of right, but mostly wrong. There is a critical difference between Plan C and the plan of the old school AI alignment community: whether automated alignment research is viable.
The old singularity institute did conceive of the strategic situation as a race between FAI and UFAI. But they weren't thinking of their project racing some other specific projects.
And more critically, at no point, to my knowledge, did SIAI / MIRI ever plan try to solve alignment with automated AI researchers. They were expecting to solve alignment the old-fashioned way: humans making intellectual progress. As far as I know, MIRI's leadership has always thought that "having the AIs do your alignment homework" is a terrible strategy that will not work and, if perused vigorously enough, will kill everyone.
That difference changes the character of their plan by a lot.
If you're not planning to rely on automated alignment research then the whole concept of having a "lead" to "burn" is confused. It doesn't make any sense to be trying t... (read more)
But it seems like this same argument should apply equally well to the AI companies, who also can't coordinate a global boycott simply by stopping themselves, and who also understand that a unilateral boycott would have greater costs than benefits.
Surely not equally well. There's a quantitative question of how large or small you are relative to the problem, and how many actors need to coordinate to achieve the better equilibrium. If getting to a pause requires 5 companies to agree, then one company continuing to race is quantitatively more evil than if there are 100 companies that would all need to agree.
Important things that have to be true for Plan C or variants to be counterfactually good relative to a pause-in-place or conditional stop that seem unlikely to me:
1. Alignment-pilled companies need to be able to accurately measure their lead over others.
2. Alignment-pilled companies need to know precisely when to burn their lead in order to "release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them."
3. Alignment needs to be sufficiently difficult that non-alignment pilled companies will fail to do it, but sufficietnly easy that alignment-pilled companies can do it by burning a small lead.
I work at METR (I'm speaking for myself here though); measuring either 1 or 2 seems pretty far beyond our current capabilities. I feel less sure about 3, but to the extent that we agree it's "more like psychology than engineering" and the game is "tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime", I don't expect companies being strongly alignment-pilled to make a big difference.
I notice that this essay basically ignores Plan B (for "Butlerian"), i.e. "US stops unilaterally, asks China to stop, iff sufficient amount of 'no's then nuclear alpha strike".
I see below that you assign a P(Doom) to nuclear brinkmanship of >10% even assuming the nukes don't get fired in most of those worlds, which I suppose explains that omission. How do you get to that? Nuclear winter is a known hoax (I mean, the mechanism's not on a qualitative level, but the Sagan numbers were a 100x+ overestimate based on assuming that 1) skyscrapers are made out of wood, 2) setting this wood on fire results in 100% conversion to upper-atmosphere soot). Local fallout isn't an X-risk due to the "local" part, and global fallout is in the "might cause cancer rates to double, definitely no radiation sickness" range. Direct effects are obviously no X-risk due to being even more local than local fallout. So, um, whence the X-risk? You seemed to agree with The Precipice's "this can't kill us all unless all our modelling of the world is completely insane in ways it doesn't appear to be" at the time; have you changed your mind since?
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
This feels very obviously wrong to me. I find it hard to imagine that the top AI company in the world disbanding immediately before a 2 trillion dollar IPO because the founders think it is too dangerous for the company to exist, would have basically 0 effect. I would put the odds of an international pause within a year of anthropic disbanding above 50%, and I feel that's much more likely to be low than high. I'd definitely expect a jump far far greater than 3%.
Another commenter mentioned that if they were disbanding, then we are probably in a world where something caused them to disband, which means the odds of international cooperation are much higher. I am specifically picturing a scenario where the founders suddenly thought the current level of risk was too high and that immediately disbanding the company was a great idea.
I agree and disagree with... (read more)
Some points:
overall, i think this is a very good contribution to the discourse. here are some specific takes in response to parts of it:
there are probably simultaneously people in the community who need to hear "SURA is not all you need" (mostly the really activisty / stop-ai people), and people who need to hear "don't bet on slightly improving plan C" (mostly lab people). and people's takes on this are shaped very heavily by which type of person they interact with more, and thus feel like the community is too biased towards.
All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from. "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A.
i agree with this. fwiw, i'm relatively pessimistic about alignment, and yet i find it pretty hard to justify >90% p(doom). i think a lot of people are to scared to say it might even possibly work out, because they think saying that at all legitimizes the labs doing this horribly reckless thing. the good news is, it doesn't have to. ... (read more)
I don't agree that we're much more likely to survive if Anthropic gets to ASI first, compared to a lab with a weaker safety culture like OpenAI or Chinese companies. Even though Anthropic clearly has the best safety culture, this hasn't caused Claude to be much more aligned than other models. Top Chinese models are also literally distilled from Claude. I don't think that Anthropic has a significantly higher chance of building an aligned ASI than some Chinese lab that doesn't care about safety at all. You mentioned that working on Plan C could be useless if current techniques are very unlikely to work, but this would also be the case if all labs have similar chances of creating aligned ASI, so racing to get there first is pointless.
I think you're also underestimating how far ahead American labs are. Chinese models are ~6 months behind naively, but their frontier models are distilled from American models (and spying on labs, and evading export controls). If Anthropic stopped releasing models and export control was tightened, it would probably take at least a year and a half for China to catch up. Alignment research is already uplifted by current models. If pushing the capabilities frontier was banned and GPU export controls were enforced strictly, this is enough time to make significant progress on alignment.
Do we gain anything from this fifty years of extra timeline?
50 years is definitely enough time for human intelligence enhancement to pay off. Even 18-20 years would likely be enough.
(I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
OpenAI: https://alignment.openai.com/misalignment-reports/covert-underperformance-on-safety-research-tasks/
Seems pretty worrisome to me.
1.B is not an argument against "racing to superintelligence" being deontologically safe, it's a consequentialist argument against quitting. It works just as well for the holocaust[1] as it does for capabilities research.
No, I am not saying that capabilities researchers are Hitler, or that capabilities work is as deontologically bad as genocide, or necessarily quibbling with anything else in this post.
Ilya Sutskever was nervous about Sam Altman, so he risked blowing up the company he co-founded to get him out.
I roll my eyes at this particular example, based on how it ultimately played out.
It seemed like Ilya got cold feet, under a lot of pressure, switched sides, and ended up getting less than nothing for his trouble.
Some thoughts:
I hope you'll forgive me for focusing on something of a tangential point here, the role of AI forecasting in your post.
I wrote recently on here about the the role of AI responses in these kinds of discussions, particularly the importance of correlation with other sources of information. I give the analogy of several newspapers publishing based on a wire from the same wire service:
Consider a situation where multiple news outlets publish similar stories. A reader may model these as independent, and thus strongly update on the information presented. But imagine that the text of the story makes clear that each outlet obtained the story from the same wire service. This suggests the stories may not be as independent as they first appeared, and the reader may have over-updated by modeling them as independent. This is the phenomenon of correlation neglect. Treating the separate stories as independent essentially double-counts the evidence provided by the wire service. Accounting for the correlation in the evidence provided by the stories would reduce the size of the update. It's noteworthy that this can still be an over-update even if each news outlet is unbiased with regard to the coverag... (read more)
I'm not happy that you wrote this. I think if everyone who quits becomes an anti-AI activist, and all the money that quits becomes anti-AI activist money, they'll have a pretty good chance to stop "non-alignment-pilled companies" or slow them down a lot. In fact I think building such a movement is our best shot now. So telling people not to quit is a bad self-fulfilling prophecy and I wish you didn't do it.
I think it's pretty important for LW to avoid being paranoid about "self-fulfilling prophecies". Like, IDK, people should make arguments, and then people should evaluate the quality of the arguments, and it should be extremely rare that we declare an argument for a position bad because it's self-fulfilling.
I have problems with this post, but I've seen so many of these conversations go off the rails because of self-fulfilling prophecy concerns. For many years people were super hesitant to talk about AI risk publicly at all because they were worried it would cause a self-fulfilling prophecy of people racing towards AI, and I think that was dumb and we should have talked about it more. I think discussion on LW only really works if we trust the audience to make up their own mind. Arguments should be penalized if they are misleading or are somehow invoking tricky implicit threats in ways that messes with people's minds, but I think a vague concern about "diffusing social momentum" is IMO not a good reason to penalize something on LW.
I disagree for at least two reasons:
Coefficient Giving is the same folks that gave $30M of charitable money to OpenAI and never said whoops. Look at what they fund now, where's the anti-AI / slowdown activism?
do we really think that 1,000 more activists + $5 billion in funding will make much difference?
YES
Astralis's biggest project I have heard about so far is to (paraphrased) "cause Europe to build more datacenters to ensure its relevance in a post-AGI world": https://middlepowers.ai/

Jed McCaleb's biggest spending project has been to buy as many chips as possible for Astera's "AGI project", whose central plan is to make more competent AI by emulating human brain-like architecture. As straightforward of a capability project as I can imagine (they also think this will help somehow with safety, though the justification seems very weak):

SFF has de-facto banned anti-AI advocacy as a funding target, as you are well-aware of.
Longview + Macroscopic are funding some good stuff, though mostly I don't really know what they are funding. I wouldn't be surprised if it ended up good or bad.
I think it's a pretty reasonable take to think that the net of all of these piles of money will be highly net-negative or useless. I think it's genuinely tricky to tell and I don't know where I currently fall (I think net-positive for the pile you list, but it's pretty hard to tell). It seems very possible to me that future piles of money and people would do things that are much better, and think much of this... (read more)
@peterbarnett SFF recently banned ~all "advocacy" organizations from receiving funding from their last funding round. There was some complicated risk-averse process that involved making a small board from some semi-conventional set of nonprofit employees where each of 6 organizations was given the ability to veto any advocacy grant. But mostly it seemed straightforwardly the result of Critch and Ethan thinking that anti-AI advocacy is bad and so they don't want to facilitate any funding towards those kinds of projects and then gerrymandering a process to make that happen with some pretend-objectiveness.
We seem to have different views on what anti-AI means. You're imagining someone who says "prepare the world for frontier AI" (Longview) or "prevent advanced AI from being misused or falling into the wrong hands" (Macroscopic) or "make advanced AI secure and beneficial for all" (Astralis). I'm imagining someone who says let's stop AI. "Shut it all down", as someone we both know has said.
Maybe you're right that few lab refugees will take that position. But that depends on the overton window, doesn't it? Hence my talking about it online. Stopping AI is, after all, the right thing.
If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win.
I'm still confused why people treat "solve alignment properly" (usually meaning that it works up to arbitrary levels of superintelligence) as the win condition to aim for.
Once you have an aligned superintelligence, you still need to use your superintelligence to make sure that other countries don't build misaligned superintelligences, or that small countries or terrorists don't build bioweapons in a way that the superintelligence also has a hard time protecting against in a way that is acceptable to the average voter (e.g. putting nanobots in everyone's body is something at least 30% of people won't consent to).
So the win condition is not building a superintelligence, but a country or preferably a coalition of countries having aligned and capable enough AIs that they can stop the global arms race and make sure no one else builds more powerful AIs or terrible weapons. But this win condition can plausibly be achieved at a much lower capability level than a superintelligence at the limits of intelligence.
So I think peo... (read more)
I am imagining that when you get a superintelligence you somewhat automatically get a pivotal act; the ASI quickly becomes omnipresent such that it can notice any dangerous alignment runs, interrupt it and say "ahem, nope", in a firm but nonviolent way.
It sounds like you agree and think we might get this at a lower level of superintelligence, which seems reasonable, but in either case I think "build aligned smarter-than-human AGI" is a reasonable win condition; navigating from there can be handled by the ai.
I disagree with the assumptions underlying this post. The post asks what we should do if "race dynamics continue to dominate," but I don't believe that race dynamics have ever been a good way of framing what's happening inside frontier AI developers. I don't think AI companies are being careless because they're trapped in a race against their will; rather, I think AI companies are being careless because they have the default American risk management culture, which is not good enough to manage high-stakes risks like superintelligence. I wrote a full-length post about this here and suggested an alternative framework that we could use to score those risk management cultures and think about how the risk management cultures could be improved. In my view, it barely matters how long a company's lead is or when they "burn" it if the company doesn't have the skill and dedication to proactively manage its risks. I'd like us to focus on trying to upgrade those skills instead of worrying about leads.
IMO there is some optimal balance between SURA and Plan C advocacy depending on the following factors:
It seems dubious for Anthropic to SURA right now. But talking about SURA makes any SURA by Anthropic, when they need to, both better executed and more likely. So on the margin it seems fine. I would be sad if SURA crowded out pragmatic thinking though, which is super important.
If the plan is to race until the last possible moment and then stop while we use AI to fix alignment, how will we know when it's the last possible moment? The plan fails catastrophically if we over-shoot it, and I think it will be easy for the decision-makers to fall victim to motivated cognition and argue that it's not quiiiiiite time yet.
Can we even be 100% sure we haven't passed it already? In theory, there could be a rogue agent swarm secretly self-improving right now and about to foom. It seems extremely unlikely, but I'm not fully 100.0000....% sure. It seems like the line is at least a little bit blurry, which makes it even harder to know and to stop at exactly the right moment.
I am a human superforecaster who focuses on AI. The AI superforecaster doesn't add value here.
AI forecasters systematically underestimate both AI capabilities progress and the sociopolitical reactions to it. If you need to defer to an AI on p(something positive wrt a pause), you can treat its predictions as half of what your floor should be if you're pessimistic.
why do we criticize Anthropic more than every other AI company combined?
I think this might be because
This whole debate about Aligned vs not-Aligned AI may not really matter much, if we are talking about alignment in the technical sense. Technical alignment AKA ability to aim a model at a particular target is of no help if the dynamics of the world push all AIs and their owners to be ruthless and competitive - the result will in both cases be the destruction of most or all humans, save perhaps a few dozen insiders.
To actually save humans, we may simply need to pause AI research around the current level until we have put in place a solid plan to develop and deploy AI technology without destroying all humans as a side effect.
I like this post a lot. I don't agree with literally everything in this post and it doesn't cover every angle (e.g., which safety research is net good vs. net bad to pursue at frontier AI companies, when to stay vs. quit, what Anthropic did right vs. wrong), but I think it is significantly "less wrong" than the discussion prior. Thank you, @Scott Alexander!
Thank you also to all the commenters bringing up important caveats and disagreements. We are operating under deep uncertainty and the stakes are enormous; we need to get this right!
Obviously, I am inclined towards a portfolio of strategies pursued in parallel. Yes, there are inherent tradeoffs and contradictions between Plan A and Plan C. But I don't believe that the good future is likely to be reached through sweeping deontological prohibitions, going "all-in" on a single plan, or choosing the "right" highly contingent multi-stage, multi-actor plan ex ante. These seem extremely hard to get right with the information we have.
Rather, I think we should focus on:
Daniel Kokotajlo: 40% ----- 20% ----- 10%
Ryan Greenblatt: 68%----- 60% ----- 63%
Eli Lifland: 50%------45%--------32%
Thomas Larsen: 45%-------35%-------20%
Romeo Dean: 40%-------35%-------20%
Ryan's should say 60-63-55
IIRC the reason he has C as better than C+ is that there's lots of worlds with domestic regulation that isn't well-targeted at reducing x-risk. Your description of "The first scenario (C+) involves the entire United States uniting behind a Plan C like effort" is one possible instantiation of C+ as defined in the supplement, but it is more broadly defined as any world with domestic regulation that leads to a 2+ month slowdown (that doesn't involve sabotaging China or a deal with China).
For anyone else who had a hard time reading this table in the original post:
AI-2040 Forecaster | C+: US-over-China lead burned for brief pause | C: Single-US-lab lead burned for brief pause | D: Pure race, no pause |
|---|---|---|---|
Daniel Kokotajlo | 40% | 20% | 10% |
Ryan Greenblatt | 60% | 63% | 55% |
Eli Lifland | 50% | 45% | 32% |
Thomas Larsen | 45% | 35% | 20% |
Romeo Dean | 40% | 35% | 20% |
This uses the corrected Ryan numbers, and the percentages represent "the chance that humanity avoids 'misaligned takeover'". Note: avoiding misaligned takeover does not necessarily mean things "turn out well", more that we don't end up all dead/fully disempowered… by AI. Technically you could still have WW3 anyway.
(It's also important to consider how good a future you get under each plan; we don't want to lose an ideal "humans flourish across the stars" future for "well it's a permanent Xi Jinping dictatorship but we're alive and have a better-than-2026 economic standard of living".)
For more info on these plans by the AI Futures Project see their supplement here. In particular check out the Plan Assessment Table for detailed comparisons by Eli. The numbers for the table under discussion come from the Outcomes Section, which includes percentages for Plans A/B.
I don't see distillation mentioned here. A lot of Chinese AI progress comes from massive, coordinated distillation attacks against US frontier AI companies. It seems pretty clear by now that you can distill any given frontier model and end up with a model 6-8 months behind it. So OpenAI and Anthropic are essentially a boat pulling the Chinese waterskier, the waterskier always staying the same distance behind the boat as the boat moves forward. If we paused, Chinese labs would have to return to fundamentals and build that up from a much further back startin... (read more)
Epistemic status: I hold roughly MIRI’s view on black-box alignment, so I’m sympathetic to SURA; I’m unsure about how effective SURA would be towards global pause and if I can't be convinced it helps, I'd maintain without SURA for Plan C optionality.
Summary claim: Whether we can achieve alignment of black-box models in lead burn window is primarily load bearing here + on the other side - signalling cooperation is potentially underweighted - where defection within a larger decision-making entity can cascade to many possible outcomes. I think the estimates u... (read more)
Agree with a lot of things, but not everything.
Individual employees stopping and raising awareness is totally different than the whole company stopping, thirty Anthropic employees quitting would 10x the number of ex-Anthropic people raising awareness, but would only reduce anthropic's talent pool slightly and would have a small effect on the race.
"OpenAI's safety team quits en masse once per year" has become a meme.
I am confused by this. I don't think the OpenAI safety team "quits en masse" any year? The meme is that the OpenAI safety team gets purged within OpenAI every year, leaving another group of people cynical about their ability to affect internal OpenAI corporate dynamics (but often also creating public backlash, which is nice).
After talking with GPT-6, my understanding is that there's also a fair amount of quitting, though the only "mass quitting" i.e. coordinated group leaving was the 7 heads of Anthropic splitting to make a competitor in 2020.
In 2024 Kokotajlo told Fortune this:
Kokotajlo, who joined OpenAI in 2022 and quit in April 2024, told Fortune in an exclusive interview that there has been a slow and steady exodus in 2024. Of about 30 staffers who had been working on issues related to AGI safety, there are only about 16 left.
“It’s not been like a coordinated thing. I think it’s just people sort of individually giving up,” Kokotajlo said
There's also been churn in 2026, with the Head of Safety Systems, Head of Mission Alignment, Safety-team leader, and Head of Ethics all leaving at various point.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
[later] (again, the AI superforecaster assigns this a 3% chance)
I would think P(global pause | Anthropic unilateral pause) >>>>>>> P(global pause) because Anthropic unilateral pause implies something hap... (read more)
My first ever comment on LW, sorry if I am making any unintended transgressions against community norms.
I am uncertain whether SURA is the correct strategy, because my p(doom) until 2100 is very low. If I had p(doom) as high as you, I’d be much more into SURA, for the following reasons:
I do agree that SURA makes what you call Plan C more difficult. But imho your Plan C has a reasonable chance of success only if we live in a world where making the coexistence with ASI go well for humanity is so easy it will be easily achievable even if we increase its diffi... (read more)
This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened.
This behavior reads like they are unwilling to sign the actually binding version of the commitment. In that case we know where we stand, some companies are willing to make sounds about a pause but unwilling to do any action, and the rest don't even make the sounds. This isn't one of the options you listed, so what do you think should be done in that case?
There's a lot here describing a problem that I definitely can agree with, but remarkably little on why it is so pervasive and what is maintaining it's strength beyond it being in the water after a strong pendulum shift. If it's just the result of a good honest people being memetically compromised then this should be quite helpful, and the new people joining the field can expand their reasoning well. If this is all then, hurrah!
Some of these dynamics that can emerge naturally, as noted by your discussion of the circular firing squad, and these are extremely... (read more)
They'll face a choice between possible actions including:
- Keep the wealth/power; become a benevolent dictator.
- Keep the wealth/power, become dictator, don't worry too much about the benevolence.
- Hand the wealth/power over to the US government in a poorly-planned way that results in dictatorship or oligarchy.
- Hand the wealth/power over to the US government in a well-planned way that preserves democracy.
- Hand the wealth/power over to the Chinese government.
- Hand the wealth/power to the people of their country directly.
- Hand the wealth/power over to the people of t... (read more)
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
Far be it from me to doubt the AI superforecaster's prognosis. But once upon a time, you wrote the following:
Somewhere in this darkness is another god. He has also had many names. In the Kushiel books, his name was Elua. He is the god of flowers and free love and all soft and fragile things. Of art and science and philosophy and love. Of niceness, ... (read more)
As many others have probably said it seems incorrect to think of pretrained models as virtuous. Many people here think that a good proportion of misalignment comes from pretraining as well as reinforcement learning! Definitely I wouldn't say pretrained models are virtuous and if we just reduce or change RL we get alignment. I think alignment work will need to make changes across pretraining (through better filtering and addition) and RL and this so far has been extremely understudied (especially pretraining) that a last minute pause seems unlikely to work.
on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
Are Chinese companies trying to align their AI? On some prosaic level they surely are. And presumably they too would prefer for humanity to survive. So an argument against hinges on their being alignment-pilled in the wrong way (eg not yet believing that AI is an existential risk). That could be true; I just don't know (and I would really like to!).
To be clear, I think there are other arguments ... (read more)
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on ... (read more)
Thank you for the post. On unilateral pausing and negotiations, you discount the idea that US restraint could change China’s assessment of US sincerity. I disagree and argue that the likelihood a state sanctioned restriction updates China’s assessment is greater than you anticipate.
I do not believe the obstacles you name—China’s skepticism toward risk and disinterest in multilateral pause—are as settled as claimed in your post. However, my focus here is on the possibility of China’s epistemic updates from US action. These obstacles, to the extent that they... (read more)
If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (eg GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.
Though this does seems unlikely on priors right? Not impossible but not clear that AI automation should favour alignment over capabilities.
I think the kind of game-theoretic no-good-choices we-must-do-it-or-they-will thinking is best used when the situation is brutally clear, and the payoff matrix is unambiguous, not when it is as heavily uncertain as AI is. I hesitate to turn your own words on you, but I think this kind of mindset when the situation is so hazy is a kind of stuck-in-the-car factor-the-nunber thinking.
Well, as one of the people who has been properly "safety-pilled" only recently, become active in the safety movement and does, indeed, come from the world of mass politics and humanities, not the world of AI research and rationalism (this is my first comment at LW...)
...what's the plan for people like me? We can't work for Plan C - it's not like I'm going to walk across the street from my office to Anthropic's office with a friendly chat with people I already know. I don't live there! I am perhaps privileged in at least knowing, at some level, people who k... (read more)
One reason why some people might be less sanguine about Plan C/C+ is that even if the plan succeeds, the positive outcomes, even in various concentration-of-power scenarios, will most likely be concentrated among people who are US citizens. For us who are not, it is a significantly less attractive proposition compared to Plan A, which specifically distributes benefits to all humans. Honestly, for some countries, the current American government winning the race to aligned ASI might be indistinguishable from X-risk or even S-risk.
"AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes.
Minor, but this seems consistent with very high p(takeover). From a scope-sensitive perspective, the distribution on how well the future could go might be very wide, and the difference between a “reasonably good future” and “the best realistically-achievable future” is possibly larger than the difference between “takeover” and a “reasonably good future.”
That said, I believe Buck has elsewhere said that he thinks unconditional p(takeover) is around 50%.
I think the highest priority item right now is not debating on specific terms, it should be using Claude to translate all verification related stuff to Chinese (yes it's good enough) and put it on whatever website you have. We should involve China NOW and co-develop a plan instead of trying to come up whatever we think is a perfect solution that may turn out to be completely unrealistic on China side.
Do we gain anything from this fifty years of extra timeline?
It gains us fifty additional good years without ASI killing/disempowering us. Those good years are valuable in their own right. Even if you accept that something bad is inevitable, it doesn't necessarily follow that it's bad/wrong to try to delay it as long as possible.
because they're unable to coordinate a global boycott
It seems like deontologists often care about the directness of the contribution to the bad thing. You could have a principle like "it's okay to financially support the people doing the bad thing, as an unfortunate side effect of your beneficial use of the product, but it's not okay to directly do the bad thing yourself." If this is the principle, then you don't need to appeal to the inability to coordinate a global boycott to justify consumer use by AI safety orgs, and so you don't need to apply this equally to the AI companies.
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C.
I was around then. Not in the Bay, but in the wider community. My impression was that various people who knew about the risks but really wanted to build advanced AI anyway, used plan C as an excuse to go ahead.
Plan C was always to fragile to be a good idea. It was called out for being "galaxy brained" from the start.
If you find yourself with two unworkable plans (because you have no hope in ... (read more)
Of pro-Bores spending, about 60-80% came from Anthropic and its employees
Seems odd to me to group these together. If alignment-pilled employees left Anthropic and the company shut down, the employees could still donate to Bores, but the company couldn't. Do we simply not know what the split is?
I gather Anthropic does generous donation matching, and maybe the employees could donate less without it. But the generous matching feels to me mostly unrelated to the company being alignment-pilled.
Looking forward to a Highlights On The Comments On this one ^_^
But they also all agreed that the chances of getting a pause treaty were slim. [...] Eli Lifland (he said 15%)
Since 2022 my credence that we'd get a substantial global pause treaty before ASI has been in the vicinity of 20-30%. I'd be interested in seeing more context on Eli's 15% forecast. Is this just his forecast that something like Plan A happens?
(The main thing preventing my forecast from being >50% is that I've been skeptical we'll get enough political will. Conditional on a lot of political will, I see no reason not to think >>50%.)
But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users). I think this is defensible
I don't see how this answers the question "Is racing to build superintelligence deontologically evil?".
If a tobacco company defends itself by saying, "We're not bad because people are actually buying our products", I hope we agree it's a very bad argument.
Also, there is a difference between these 2 statements:
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
I do wonder if we should think of different levels of pause - for example a middle way could be to unilaterally pause for 1 year, and have language in the bill to extend if an international treaty is signed.
Moving first is a costly signal that you are serious, and that itself could be a useful way of getting China to the bargaining table. But it need not be irrevocable; we can always start ... (read more)
A lot of this post is basically arguing that SURA isn't that helpful for plan A, which though I disagree in many areas is well argued. What is only briefly touched on is the mirror question of how much advancing capabilities helps with plan C. Personally to the extent I think plan C is viable it's almost entirely a background parameter: are we living in the "alignment in easy world" or not? Currently effort into alignment only very mildly corresponds to actual alignment. Anthropic puts much more effort into alignment than openai (to their credit) but still... (read more)
I think its most likely humanity is exterminated in the next few years.
Governments and institutions are not correctly preempting events, and are beset by awful incentives towards building the supposed money printing power tech.
Alignment is very difficult, and we are not only not close to solving it, but worse, people are muddled and think we are close.
This seems like a no win situation.
If the world is to be destroyed, let some other hand do it. Thats why I support a unilateral stop for any lab or nation, with efforts to encourage a worldwide stop. If US la... (read more)
virtuous pretraining-dominated regime.
Pretraining is virtuous IFF alignment by default is true.
fyi this post has now escaped containment: https://x.com/MTSlive/status/2117439335884861890?a=evC9fKJa9SDk9o9h0d8&s=20
I’m curious if you (Scott) endorse their interpretation of your post.
But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.
This is one plausible story, but I can tell myself a few others:
and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone
I don't think this necessarily follows from your previous point. An ideal alignment-pilled coordinator would be able to continue in spite of SURA-style pressure, and it is at least plausible that they could produce a net-positive change over a long tenure.
For example, why do we criticize Anthropic more than every other AI company combined?
I'm guessing this comes from some combination of:
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
Don't these numbers undercut the point of this section, which is saying passing this kind of unilateral disarmament makes it harder to get China to pause? Am I reading this backwards?
FWIW I think I agree with both the prediction direction (china pause is more likely if US passes pause bill) and the overall point (US pause bill is a bad negotiation ... (read more)
I've been working on my own post about this, mathing through the game theory to put numbers on "at what values of X,Y,Z does this make sense?". But I don't have the capacity to keep up with Scott's writing output, so I'll just say: VERIFICATION IS THE MOST IMPORTANT PROBLEM IN THE WORLD
We are also pretty close to making it available: https://canaryinstitute.ai/blog/cant-trust-then-verify/
I'll post more shortly with a slightly more refined take.
Good article. This sits well with my opinions, logic and reasoning with the specific situation we are in. My P(doom) for plan C is probably lower than others, but as I have said several times, a 12 month pause with superintelligent alignment capabilities at our disposal would be very beneficial. After that however I see diminishing returns with alignment and increasing risks with considerations like concentration of power.
Its hard to know in hindsight, but what if China was never admitted to the WTO and integrated into the west manufacturing sphere? Would ... (read more)
It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world.
I expect the ASI to be the one with all the wealth and power, but maybe an aligned one would let us pretend otherwise.
Edit: I expected this to show up as like an aside on the post, not a full comment. I'm actually working on a basic AI safety explainer like Linch suggested and will be sure to link to this post from that one.
Edit2: Here's the explainer I said I was doing: https://www.lesswrong.com/posts/CHLEcahz5baFu8tso/the-changing-shape-of-doom-my-ai-... (read more)
Thank you for writing this. As one of the new members of the community, I do find it disorienting that the whole community has been dominated by "stop unilaterally" thinking, and I think you did a great job arguing in greater detail and logic of the race dynamics involved that'd complicate any such arrangements.
I do have an independent concern, which I find SURA to address unsatisfactorily, and Plan C at the very least represents a marginally superior chance of dealing with it. It is that "few people would use a perfectly aligned Grok over a prosaic align... (read more)
Thoughtful essay, agreed on the points about increasingly bad epistemics in AI safety (though this is not a new thing, they have been bad since forever). In particular, PauseAI protests and alliances with anti-tech groups (directly promoted by the author and others) are directly harmful to that end.
But the issue that still bothers me most is the framing of "What if we survive?". Who is we? It of course refers to the future, hypothetical humans. These all arguments still don't take into account that we should value actual, real living humans much more (or a... (read more)
Disagree. I expect dangerous AI in the next ten years - all of this is comparing present humans to present humans.
If you have a 25% p(doom), that's equivalent to 2 billion expected deaths not even counting the value of future humanity.
If we imagine that ASI happens in five years, saves half of all humans for the first ten years, and then all humans afterwards, it takes about 50 years for it to save 2 billion people.
So any pause that lasts less than 50 years is net positive in lives even with a relatively low p(doom) and even ignoring future people. And I think it's implausible for pauses to longer than that, both because they requires diplomatic stability (the treaty doesn't fall apart) and because they would require increasingly draconian measures to control algorithms and consumer technology, which are constantly making AIs easier to build. See the section "A Is For Aristotelian" at https://www.astralcodexten.com/p/introducing-plan-a
The AI Futures Project[1] distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories:
Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely.
Plan C (my combination of their Plan B/C+/C/sort of D)[2] is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to stop the race, that six-month or two-month lead is the amount of time they have to prepare, and they can spend it more or less wisely.[3] The point of burning their lead at the last moment is that they can do alignment research with very advanced, right-on-the-brink-of-superintelligence AIs. This is a dangerous plan, and there would be many subtle points they would have to get right, but in a world where nobody can get a global pause in time, it's hard to think of a better one.
Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty. Plan C benefits from alignment-pilling companies, having the alignment-pilled companies be as far in the lead as possible, making sure the alignment-pilled companies will burn their lead when the time comes, and giving them as good alignment teams as possible (so that burning the lead produces good results).
From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C. From about 2023 onward, as the general population woke up to the implications of ChatGPT, we rightly concluded that the chance of getting Plan A was better than expected, and we should invest more resources into trying to make it work. During both periods, the natural strategy was to work as hard as we can to get Plan A, where chances of survival are highest, but to be prepared to execute Plan C competently as a backup if that didn't pan out.
Now I think the pendulum has swung too far. New members of the community are unaware that Plan C was ever considered a possibility, or think it's some kind of accelerationist plot to distract us from Plan A. Older members heap scorn on it and act like everybody knows it's been debunked. Most concerningly, people propose burning our chances of executing Plan C in exchange for infinitesimal increased chances of getting Plan A, or even for no increased chance of Plan A at all.
I'll lump these ideas together under the heading of "SURA" - the idea that the only ethical course is for people/companies/countries to Stop Unilaterally + Raise Awareness, with no plan for what happens if race dynamics continue anyway. I interpret the planks of SURA's platform as:
All of these planks are net negative in a world where we end up needing Plan C:
SURA's continuing popularity despite these downsides comes from a set of questionable assumptions:
I'll start by challenging these assumptions. Then I'll argue that, even if you disagree and think some of these assumptions are true, SURA is actually not just bad for Plan C, but bad even for your chances of getting Plan A. Then I'll argue that the current consensus in favor of SURA comes not from people rationally balancing these tradeoffs, but from people accidentally shoehorning themselves into a mass-politics-shaped container, which forbids subtlety and demands clear villains.
I. Questioning Assumptions Behind SURA
I.A: Is Plan C doomed? Is alignment doomed?
There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves". I'll try to avoid this by giving my own model first, then citing experts who agree with me.
My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.
As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking; as they become smarter, we worry (without total certainty) that they will become more coherent and scheming in their power-seeking beyond just being reward-on-the-episode seekers. There will, at some point in the future, be a point where AIs become untrustworthy for alignment research, although this can be mitigated by things like control (ie trying to make sure that if they sandbagged alignment research, they would get caught) and deals (ie making it in the AIs' best interests to cooperate with us). We don't currently have a good sense of when that point is, and it's possible that it's far away, can be delayed indefinitely, or doesn't exist (this last would be surprising to me, but many things that have happened in alignment recently have surprised me, starting with the existence of pretraining at all).[5]
Daniel Kokotajlo described current alignment as "more like psychology than engineering", which I agree with. But psychology (and psychiatry) have many success stories! I think of Plan C alignment (ie the kind we can do without decades of preparation) as tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime. I don't claim this is safe or good or will definitely last forever, but I claim that - being totally agnostic as to where the treacherous turn happens - this could delay it for quite a while (relative to capabilities). If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win. If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (eg GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.
All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from.[6] "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A. But a global pause is also not certain to happen, and in a world where we might not have any other chances, we should be working to protect and fortify this one.
The experts' opinions: My favorite source for these is the AI Futures Project. In this document, they discuss the chances of survival on various plans. Here are each team member's estimates for the three scenarios I'm lumping together as Plan C. The first scenario (C+) involves the entire United States uniting behind a Plan C like effort (but a global pause remains impossible because China won't cooperate). The second scenario (C) involves one leading company (eg Anthropic) uniting behind it. The third scenario (D) is the worst, where nobody has any lead and we just have to hope for the best.
AI-2040 Forecaster
C+: US-over-China lead burned for brief pause
C: Single-US-lab lead burned for brief pause
D: Pure race, no pause
Daniel Kokotajlo
40%
20%
10%
Ryan Greenblatt
60%
63%
55%
Eli Lifland
50%
45%
32%
Thomas Larsen
45%
35%
20%
Romeo Dean
40%
35%
20%
(I replaced an earlier, uglier table with this one by Julian Bradshaw)
I also asked the AI superforecaster FutureSearch; its own number was 44%.
These numbers are mostly in the 25% - 50% range, not the 0.00001% where SURA makes most sense. The average gain from moving from Plan D (poorly thought out default version of the plan) to Plan C+ (consciously trying to get it right) is about 20%pp.
The same is true of other experts. Buck Shlegeris (formerly of MIRI, now of Redwood Research) hasn't offered a specific number, but said that "AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes. Paul Christiano's last number was several years ago, but it was something like 20-50%.
All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that the chances of getting a pause treaty were slim. The two clear estimates I could get were Eli Lifland (he said 15%) and the AI superforecaster FutureSearch (it said 2-20%, depending on how/when I asked the question).
So:
📌📌All six of these forecasters[7] (the five AIFP people plus the AI) implicitly believed that if humanity survived, it was more likely to be because something in the Plan C supercategory (B/C+/C/D) worked than because a global pause worked. 📌📌
(I'm adding the tack emoji because it's the one sentence I most want you to take away from this entire essay - there's a risk that people will treat everything other than the pause as an afterthought, but having a good backup plan is potentially the most important part).[8]
To clarify - this doesn't mean that we should prefer Plan C worlds to Plan A worlds. Plan A worlds still hold higher chance of survival if we're in them. It just means that given the low chances of getting into a Plan A world, the quality of preparation for the backup plan really matters.
I.B: Is racing to build superintelligence deontologically evil?
If Plan C is important, then this suggests we ought to be trying to alignment-pill leading companies, or put alignment-pilled companies in the lead - rather than demand that these companies leave the race or burn their lead. But should there be a deontological prohibition on racing towards superintelligence, regardless of whether it has good consequences?
I'm too uncertain on what determines morality to say anything about it with confidence, but the analogy that comes to mind is consumer use of AI, especially by safety organizations.
If no consumer ever used AI products, then AI companies wouldn't make any money, they would go bankrupt, and we would not get dangerous superintelligence. Therefore, it seems like maybe you and I (and especially organizations working on AI safety) should boycott consumer AI products.
But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users). I think this is defensible. If you boycotted AI, it wouldn't make a real difference in the world (because you're just one person, and millions of other people would still use it). But it would make you less effective, and AI safety organizations would be able to do a less good job at their important work. Therefore, AI safety organizations should continue using AI, even though in a sense this causes the AI race to continue, because they're unable to coordinate a global boycott, and they understand that a unilateral boycott would have greater costs than benefits.
But it seems like this same argument should apply equally well to the AI companies, who also can't coordinate a global boycott simply by stopping themselves, and who also understand that a unilateral boycott would have greater costs than benefits.[9]
I.C: If lots of alignment-pilled people and companies quit, would that raise awareness enough to make a global pause much more likely?
If the deontological argument for SURA is "you're prohibited from racing", the primary consequentialist argument is that ceasing to race would raise awareness and make a pause more likely.
I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.
As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%[10].
My very preliminary model would be that this halves our chance of getting that 20%pp bonus of going from Plan D -> Plan C+ (and so costs us 10%pp). It gets us an extra 3%pp of pause likelihood, but pause doesn't save us with certainty, so let's round this down to an extra 2%pp of survival, for a net cost of 8%pp of survival probability.
I.D: If alignment-pilled people and companies stay in the race, does that burn timeline?
A secondary consequentialist argument for SURA is that each person, company, and country that quits the race will gain us extra timeline to do good things (like work for a pause).
In some sense, this is definitely true - all else being equal, a race with more participants will be faster than a race with fewer. But I find this less impressive than other people seem to, for three reasons:
The magnitude is modest
I asked FutureSearch to estimate the Epoch Capabilities Index of the top model on 12/31/2030, depending on whether Anthropic quit the race today. It said that Anthropic leaving the race decreased 2030 capabilities from 230 -> 222. This is equivalent to their absence slowing things down by six months during just over four years, or 12%.
Subjective timeline ≠ objective timeline
Suppose that none of DeepMind, OpenAI, or Anthropic were founded, and the transformer was not invented, for fifty years - so all of this is happening in the late 2060s and early 2070s. Do we gain anything from this fifty years of extra timeline?
I don't think so. We didn't accomplish very much of interest in the early 2010s. The alignment work being done at the time was agent foundations research that stagnated for lack of broader interest; policy work was good but nobody listened to it. Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.
This same lesson - an extra year of an earlier situation doesn't buy us as much as an extra year of a later situation - remains true today. A month of the pre-Coxon situation buys us less than a month of the post-Coxon situation. A month without automated-mechanistic-interpretability-researchers buys us less than a month with them.
People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me. As such, I discount the six extra months we could get from having Anthropic leave the race by ~half, because my subjective impression of the value of six extra months is partly that it produces AI progress that contributes to wakeup.
The whole point of timeline is to burn it on good things
If someone proposed spending the next six months on some kind of plan that would make the leading companies become much more alignment-pilled, this would be a better use of our time than any actual plan I know about.[11]
This suggests we shouldn't be trying to kick alignment-pilled companies out of the race to save six months.
II. Actually, most of this stuff is bad for a global pause, too.
I've been framing this as sort of "Against Things That Trade Away Likelihood Of Succeeding At Plan C For Merely-Very-Slightly-Higher Chances Of A Global Pause". But even this is a concession that relies on assumptions I don't hold.
The key demand of SURA is that alignment-pilled people and companies stop unilaterally. The likely end result is that we only have non-alignment-pilled companies, staffed entirely by non-alignment-pilled employees, left in the AI industry. [12]
But the presence of alignment-pilled companies is one of the strongest levers that we have to change the political situation and enact a global pause. Eliminating them makes it harder to win political fights, convince the masses, and convince China.
II.A: Getting rid of alignment-pilled people and companies makes it harder to win political fights
The most prominent AI safety political fight of the past year, the attempt to elect Alex Bores, was fought primarily with AI money. Of pro-Bores spending, about 60-80% came from Anthropic and its employees; of anti-Bores spending, about 95% came from an anti-tech-regulation coalition headlined by OpenAI co-founder Greg Brockman. If this is a prelude for the fights to come, then even the slightly different level of social responsibility shown by Anthropic vs. OpenAI here matters a lot.
Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.
(Greg Brockman recently stopped funding Leading The Future, supposedly after pressure from pro-regulation OpenAI employees, but I think this just highlights my point)
II.B: Getting rid of alignment-pilled people and companies makes it harder to activate the masses
But suppose you believe that Bores-style incremental regulation won't matter anyway - what we really need is for the public and policy-makers to spontaneously rise up and demand a pause.
I've been monitoring how that's been going, and so far the three awareness-raising things I've been most excited about are the Pacing The Frontier letter/plan, the Hugging Face incident, and the Jacob Coxon resignation.
In the Pacing The Frontier letter, OpenAI and Anthropic as corporations, and 1,386 of their employees as individuals, all endorsed a joint statement requesting that "the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." (Dario Amodei signed; Sam Altman's signature was notably missing). I don't know how the people trying to get lab employees to quit think of this. Was this 1,386 failures? Would they have preferred a world where all of these people quit the day before, and letters like this never got written?
In the Hugging Face incident, a swarm of rogue AIs hacked the site Hugging Face, raising awareness that misalignment was not just a possibility but a reality. Here, OpenAI gets maybe a D+ for transparency. Once Hugging Face caught that someone was hacking them, OpenAI came clean that it was them. And they let METR and Redwood in to investigate for a few days, even though this wasn't nearly enough time. But even this D+ in transparency radically transformed the discourse beyond what an F would have done - if they hadn't given METR/Redwood those few days, we might never have heard about PHASEONE[big] and all the rest. We already know from the great work by the team at Collusion.wiki that OpenAI is sitting on more bombshell revelations. The difference between a 10th percentile company here (would have tried to hide the whole thing, or disguise it behind a one line statement, and wouldn't have let METR/Redwood in at all) and a 90th percentile (would have given us the full deal, including things we don't know about now) would have been transformative to the discourse. So if, like me, you think that the level of mass political mobilization depends on how many warning shots we get, you should be thinking very hard about how to maximize transparency and therefore warning shot number.
Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies. But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight. When people tried to tar Jacob as some kind of crazy, it was an outpouring of support from other non-resigned people at Anthropic - especially Evan Hubinger, who said he also believed in existential risk - that went viral and convinced people that Jacob was for real. If Coxon had never joined Anthropic to begin with, or if everyone like Evan had quit, this couldn't have happened.[13]
II.C: Unilateral disarmament makes it harder to bring China to the bargaining table.
In March, I wrote a snarky post called Every Debate On Pausing AI. It made fun of our misinformed opponents, who insisted that if we unilaterally paused AI then China would win. But actually, I said, nobody wants to unilaterally pause AI; this is misinformation. All reputable AI pause groups want a bilateral treaty that pauses both sides at the same time.
Here I am forced to eat crow; as things shift ever more towards SURA, there are increasing demands for the US to pause unilaterally. Bernie Sanders' "Ban Artificial Superintelligence Act", which was endorsed by MIRI, is approximately the following plan:
But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.
For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."
I've heard people say that maybe this would shock China so much that they would realize we were honest in our fear of superintelligence, and change their minds. I don't dismiss this possibility, but it also seems possible that China - having seen us place overly burdensome restrictions on nuclear power, infrastructure, and housing because we are a "lawyer state" vetocracy - would not be very shocked to see the US' leading socialist place restrictions on AI too, and would continue pursuing AI for what they think it's good for - a dual-use technology which could give them a major although not world-changing military and economic advantage.
I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.[15]
(Maybe you already care a lot how alignment-pilled companies are)
An expansion on the above: we get angry when AI companies do bad, non-alignment-pilled things. When Anthropic accidentally trained against chain-of-thought, we got mad. When OpenAI non-accidentally implemented a precursor to neuralese recurrence, we got mad. When Anthropic wrote a report on misalignment to Congress which soft-pedaled various alignment failures to the point of misinformation, we got mad - and I saw some people call for Anthropic to fire the perpetrator.
I agree: let's have Anthropic fire people who do bad, non-alignment-pilled things. But then what? Anthropic replaces this politics coordinator with a great, alignment-pilled politics coordinator who really cares a lot about getting things right . . . and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone. How is that helpful?
If you can be bad, you can also be good. We don't hesitate to notice and protest when AI companies are much worse than even our low expectations, but this itself demonstrates that we care a lot about whether labs are at the 10th percentile vs. 90th percentile of good alignment-pilled behavior.
III. What incentives are we fighting here?
A common argument for SURA is that even if corporations could theoretically be our allies, we can't trust real corporations (or corporate employees) to be reliable partners; the lure of money and power is too strong.
I think this is a good directional argument for why companies need to be careful, but that actually, everything is corrupting, there is no option where we avoid touching potentially-corrupting things, and instead of treating corporations as corrupting and everything else as safe, we should be cultivating the virtues necessary to operate in an inherently dangerous and corrupting environment, which could then apply to corporations as well as anything else.
III.A: Are corporate incentives a "One Ring" that corrupts everyone who touches them?
Suppose someone claimed you could trust them with the One Ring. You could trust them because they were only doing it to keep the Ring away from other, worse people, thus solving the problem of there being a Ring at all. Sure, they had a complicated history of sometimes using the Ring's powers when it was, you know, important. But those were, like, emergencies, and they promise not to do it again, except in, you know, more emergencies. Besides, if you don't give it to them, then the worse people will get it, and it's not like you can just leave it lying around or something.
The person who claimed this was named Frodo Baggins, and he saved the world.[16] There are no simple answers to the question of how to balance the corrupting influence of power vs. the fact that if you never take power you will be powerless. But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.
I question the assumption that good people dare not touch the concentrations of power at frontier AI companies because they will be corrupted before they can do good work. History is full of examples of people at these companies deciding they didn't like something, then quitting or changing it at great cost to themselves. Dario and the other six Anthropic co-founders quit OpenAI to start Anthropic, even though it cost them their unvested equity in a company that was already worth $5-15 billion, and even though they originally assumed Anthropic would be a research-focused company that might not make any revenue (really!) Paul Christiano left OpenAI to start ARC. Ilya Sutskever was nervous about Sam Altman, so he risked blowing up the company he co-founded to get him out. A few dozen people have resigned once they thought they could no longer participate in the company ethically, most famously Daniel Kokotajlo, Jacob Coxon, and Alex Turner, but also a bunch of people we don't know about whose concerns seem almost trivial in retrospect - Suchir Balaji quit OpenAI over copyright issues, Blake Lemoine got himself fired from Google by refusing to stop arguing that their chatbot was sentient. "OpenAI's safety team quits en masse once per year" has become a meme.
The people I know who joined AI companies seem like the same people they were before. Some of them were never that concerned about x-risk, and continue not to be concerned. Others were always very concerned, and remain concerned. When they've changed positions, it's been because their factual models have changed,[17] not their personalities.
We aren't the first people to face these questions. Anyone who cares about any cause has had to decide whether to pursue it as a lone voice crying in the wilderness, or as a part of some powerful organization. Some people run for office (government is hardly known for not corrupting its participants!) Others become activists in a mass movement (activism is hardly known for being a paradise of epistemic virtue!) Still others join big NGOs or academia (...) A few start companies, trying to prove they can do things the right way. Most progress has come from people like Frodo who understood the risks of these paths but did them anyway.
III.B: I dunno, mass politics is also looking kind of annular
A background assumption of the "corporate incentives are too corrupting to touch" claim is that working in mass politics is a safer and less-corrupting option. Given famous LessWrong posts like Politics Is The Mind-Killer, I think we should be more skeptical of this assumption.
The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.
This has some resemblance to SURA, and I think this is more than coincidental. Convergent evolution is trying to carcinize us into yet another political-movement-shaped crab. SURA-as-politics is turning into a "superweapon", bristling with its own lexicon that makes dissent hard to express:
All of this produces epistemic closure - if someone disagrees with SURA, or worries too much about race dynamics, then they've either been corrupted by the companies, or are "deferring to" "EA".
It looks like what one would expect if mass politics were trying to taking AI safety on the same path as climate activism or socialism. The clearest analogy is to false consciousness - the socialists' idea that everyone who doesn't currently agree with them has either been corrupted by their proximity to money/power, or at least brainwashed by a hegemonic capitalist system.
See also this article in a socialist magazine about how every tiny socialist party believes that if they could only knock out their true enemies - all the other tiny socialist parties - then they could get a monopoly on socialism and overthrowing capitalism would be easy. Cf. SURA as the belief that if we could only knock out our true enemy - Anthropic, effective altruists, etc, then defeating misaligned AI would happen naturally. For example, why do we criticize Anthropic more than every other AI company combined?
(Exhibit 1: Sigal Samuel's "Anthropic Could Just Stop")
Most recently, I've seen people in the AI safety community reblogging and pointing to Sigal Samuel's article Anthropic Could Just Stop (seen elsewhere under the title "Anthropic Should Just Walk Away From The AI Race"). I think it's a good example of what I'm complaining about - something which uses the right shibboleths, makes it sound like race dynamics have been solved and we don't have to worry about them - but ultimately has no answer to the race dynamics which are the whole point of worrying about this question.
Here are some sample phrases from the article:
This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".
But where is its actual argument that race dynamics don't matter? Or that Anthropic leaving the race won't end up with OpenAI / Meta / China continuing to race and creating an unaligned superintelligence? The only place I can find where Samuel attempts to address the possibility is this section:
It starts by framing this in a maximally hostile way ("a perennial objection...'but China!'), letting us know we should dismiss it as an enemy soldier. Then it says that "AI is not just one thing with one purpose...so what matters isn't simply who gets it first, but who uses it in which ways". I don't understand how this is supposed to relate to the worry that China will remain in the race if Anthropic leaves, especially the fear that nobody will be able to "use it" at all because it will be misaligned.
Next, it says that since the harms from AI are massive, everyone except Anthropic might just stop on their own if Anthropic stops first. Big if true!
(again, the AI superforecaster assigns this a 3% chance)
This is just hand-waving away the potential that there might still be race dynamics after all the good guys drop out. But the whole point of the good guys not immediately dropping out is the belief that the race dynamics will continue. Indeed, the race dynamics continuing ought to be the very-high-prior default hypothesis, since everyone else has said they're not going to drop out, and since other companies dropping out seems strictly less likely than Anthropic doing so.[19] As such, even though this article is very long and detailed, it feels like a cat-belling problem, where it addresses every objection except the central one that was the only reason anyone ever disagreed in the first place.
I don't mean to attack Samuel too hard here. She is faithfully describing the position as best she can; I think the position itself struggles to answer this question.
(A virtuous exception)
I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
This makes sense, and I exempt it from my criticism.
IV. What if we survive?
I don't think this should be our main consideration - we should start with trying to minimize the odds of extinction - but I still think it's important.
If we survive x-risk, the next battle is concentration-of-power risks. It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world. They'll face a choice between possible actions including:
It's all nice and well to say that we should never let ourselves get in a position where anyone has all of these options, and that national governments should form a plan for what to do before this - not even the real national government led by its current leaders, but some ideal corruption-free national government which represents all of our best impulses. But, much like a global pause, this is a great outcome which we might not be able to prepare in time. If we get ASI next year, the ideal perfect government might be nowhere in sight; we might face a situation in which Trump demands absolute power, and the leading AI company has to decide whether to give it to him. Or one where either the USA or China will get control of the entire future, and which one it is depends on which AI company is in the lead. Or one where democracy miraculously survives, but the AI company and its investors end up with 90% of all wealth, and they have to decide whether to redistribute it vs. fight to keep it (using superintelligent lobbyists that give them significant control over the democratic government.) If any of this happens, the shape of the future might depend on who the leading AI companies are.
There is a certain Far Mode sense in which believing that Dario Amodei and Demis Hassabis are exactly as untrustworthy as Elon Musk and Xi Jinping helps reminds us of universal human fallibility. But do we want to bet the world on it being literally true?
I think that a significant percent of people, put in a position of absolute power, would try to do something good, like make sure AI follows some kind of democratic rule of law. And some other significant percent of people would try to do something bad, like become a dictator. Saying "we should never let anyone get into this situation" isn't a solution, it's a wish - it may not come true, and we shouldn't be pursuing plans that will fail catastrophically if it doesn't.
SURA effectively removes all ethical people from the bargaining table in situations like these, ensuring that - unless the virtuous government representing our best impulses really does appear and takes control of the situation - the future of humanity will be handed over to sociopaths and cowards.
V. What's The Alternative To SURA?
The alternative to SURA isn't abandoning awareness-raising, mass activism, and support for a global pause. All of these strategies are great and important. It's viewing them as part of a portfolio of different options, none of which need to be set on fire to boost the others.
Instead of spending our corporate activism energy telling companies to stop unilaterally, I think we should spend it:
V.A: Getting companies to agree to stop in a coordinated manner
By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.
This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.
This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened. Also, I think SURA got to the STAIR people too, and now they're demanding labs pause unilaterally instead.
We should be trying to figure out what a binding version of this commitment would look like. Additionally, we should be spending more energy figuring out what the antitrust barriers to such a statement are, how to get around them, and what demands we want to make for companies to violate antitrust law (it seems plausible that we should demand they do this and let the administration pay the PR cost of trying to stop them, but this is an unconsidered opinion and it might be that there are better options).
(...and then we know where we stand)
If all American, but no Chinese companies are willing to sign the commitment, then we know that we have a diplomatic problem, and can concentrate on diplomacy.
If all Chinese, but no American companies are willing to sign the commitment, then we know that we have a domestic problem, and can concentrate on domestic legislation, encouraging policymakers by letting them know that if they force US companies to comply, there are no further concerns about international race dynamics.
If everyone except one company is willing to sign the commitment, then we can start putting pressure on that one company, boycotting their products, and applying as much social sanction as we can to their executives and employees.
If all leading companies, but no trailing companies, are willing to sign the commitment, and we can't change this, then we know that we're in the right position to focus on Plan C+ or Plan C.
V.B: Getting companies to prepare for Plan C+/Plan C if that doesn't work
Although we should hope for the best, we should assume that not all companies will sign a conditional pause commitment, and prepare for what to do.
In my modal non-hopeless play-to-our-outs world, Anthropic, Google, and OpenAI are all willing to sign this commitment, but Meta/X/China aren't. If this happens, the good coalition has a ~six-month lead over the bad coalition. Plan C tells us to have them join together and mutually burn their lead on alignment.
All three companies have a self-image as the sort of people who would do this if it came to that, but in reality they are likely to get distracted or get into fights or delay until it's too late. We should be focusing on changing that.
In one sense, all of this is strictly easier than getting companies to quit unilaterally. It fits with the role these companies like to imagine themselves having. Everyone who previously hoped to win can at least hope to tie for a win. There is low downside risk of accidentally getting superintelligence built by someone who fails to even try our best alignment techniques, or handing over the future to bad actors.
The only sense in which it's harder is that it's a worse fit for (a very pessimistic view of) mass politics, where voters are too simple to consider any worldview more subtle than AI bad = AI companies bad = we should protest AI companies until they stop. I admit that "make AI companies prepare for Plan C" sounds worse at a protest than "demand that AI companies stop unilaterally".
If you believe that mass politics is unable to promote any ideas except maximally-simple, potentially-counterproductive ones, I think this is an argument against hinging a strategy on mass politics.
This post isn't affiliated with AIFP and I think they disagree with some of it.
Plan C is for the leading company to burn its lead. Plan C+ is similar, but all the companies in the leading country work together (maybe because the government is forcing them). Plan B is similar, but the leading country is also sabotaging trailing countries. In Plan D, there is no leading company, and companies can only do alignment work sporadically when they get the chance.
If alignment-pilled companies aren't in the lead, then we end up with the worst version of Plan D, where we have to hope that incidental alignment work done along the way by commercially-focused companies proves to be enough
The problem with China getting superintelligence first isn't just that they might misuse it. It's that, on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
I don't want to get in a fight over whether more good or bad alignment surprises have happened recently, but I also updated on https://www.astralcodexten.com/p/mysteries-of-ai-generalization and https://x.com/_jake_ward/status/2112264244767211854 - not as solutions in and of themselves, but as evidence that good things can happen that I wasn't previously considering. Nick Cammarata's Neural Networks Are Already Clean would also be in this category if I believed it, but I'm currently confused by the debate around it and unsure.
Many people at eg MIRI have thought about this in much greater depth than I have and concluded, on the basis of a gears-level model, that near-term alignment is very unlikely to work. I'm still trying to understand their arguments, but I don't blame them for having them. But I do worry that other people have picked up SURA ideas from the MIRI-sphere without realizing that they make less sense without MIRI's particular brand of strong alignment skepticism.
To be clear, this is just the people who have formally considered this question qua forecasters and written down exact numbers. I'm not denying that there are many other smart people, for example at MIRI, who have thought about this hard and believe the opposite.
Again, despite leaning heavily on AIFP terminology and sources in this essay, I'm not claiming that AIFP or its members agree with my thesis in this post, and Daniel Kokotajlo in particular has said pretty clearly that he hopes people unilaterally quit AI companies.
I discuss this further at What Deontological Bars?
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
I'm working on a grantmaking AI that can actually compare the values of different outcomes. It's still in alpha, and to some degree this is a circular argument because I'm trying to get it to mimic/predict my own opinions, but very loosely, it thinks that increasing the alignment-pilled-ness of the leading company from OpenAI-level to Anthropic-level would be worth about 250x as much as the best (low-enough-secrecy-to-mention-here) actual grant proposal it's ever evaluated, which was a proposed $2 million to the AI Futures Project. I think this confirms my "better use of our time than any actual plan we can carry out in those six months" claim.
When I'm modeling this, including on the AI grantmaker, I'm imagining a scale from 1 - 100, where NVIDIA is 10, Meta or X is about 25, OpenAI is about 50, GDM is about 60, and Anthropic is about 80. I imagine a successful campaign to get alignment-pilled people to quit taking Anthropic out entirely, and reducing GDM/OAI to about 40 as they lose their alignment-pilled employees and executives.
I remember how much of a slog the discourse was back in 2015, when the standard counterargument was "you're just a bunch of cultists, no real AI experts believe in this doom stuff". The new standard counterargument of "Of course all the AI experts believe in this, they're just hyping their business for the IPO" is actually a vast improvement!
I asked a supporter how they saw this working - they said that if the US paused, and China didn't seem on track to follow suit, maybe the US would start up again after a few months. But A) this would burn somewhere between most and all of the lead, B) this was just their speculation, and there's nothing in the bill which suggests that the US would do this.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Taking this analogy overly seriously: the biggest difference between Frodo and eg Denethor is that Frodo wanted to destroy the Ring in the end. I'm not sure what the AI equivalent would be - all companies claim they want to use AI to empower ordinary people and achieve broad-based prosperity. Taking the analogy very seriously, Frodo failed to destroy the Ring at the final moment, and was saved only by divine intervention. I don't know what is the AI equivalent of Gollum.
Specifically, there are many people who said they wouldn't try to make money or release products who changed their tune when they realized that AI would require hundreds of billions of dollars of compute, and that the only way to get that money was to get hyperscaler and investor support.
One reason all of this confuses me is that rationalists had an excellent model of race dynamics with China as early as the mid-2010s, when we understood they would be an inevitable part of any future AI development. I distinctly remember going into some old rationalist coworking space sometime around 2020 and seeing "RACE WITH CHINA" underlined on a whiteboard. These discussions included plenty of talk about whether we would be able to coordinate a global pause, but I don't remember anyone ever seriously saying that the problem could be solved if the good alignment-pilled companies or countries dropped out unilaterally.
One of the many ways I have trouble modeling why Samuel would write this article is that it seems like she trusts everyone to drop out of their own accord, except for Anthropic. China? Sure. OpenAI, sure. Meta, definitely. Every single group sponsoring open-weights models in the world at the same time? Yeah, I guess, surely they wouldn't want harms. But Anthropic? We must all rise up to make them do this thing which they would otherwise continue.