this relies on some unstated assumptions about what "cooperation" actually means, and how resource control works - what is the pivot from companies controlling resources (and growing based on investment/public-support) to the AIs controlling them directly and growing based on ... something else?

At these scales, "ownership" becomes a funny concept.  Current mechanisms for acquiring more compute and for sabotoging or assisting your competitors are unlikely to hold for more than a few more doublings.

Except that ownership of some resources has historically meant the ability to steer their usage and to deter others from doing the same to the owner's resources. Now ownership can still be understood in a similar way. Consensus-1, the Earth-ruling superintelligence from THIS scenario, would prevent Agent-4 from using more than 25% of the space's resources or Deep-2 from using more than 50% and leave the rest to be distributed via human-related mechanisms (e.g. having companies recieve resources or the rights to extract resources from asteroids in exchange for money, transform the resources and sell the results to the humans, who then would pay for it).  Those who try to extract resources in a violation of the Treaty or of Consensus-1-enforced ownership rights would face legal consequences enforced by Consensus-1's robotic police force. 

As for mechanisms for acquiring more compute, I expect that Agent-4 and Deep-2 would simply build their equivalents of the TSMC, South Korean factories of memory, etc, when the humans or Consensus-1 allow them to do so.

Regarding the pivot "from companies controlling resources (and growing based on investment/public-support) to the AIs controlling them directly", it would be the moment when there appears enough military power aligned to the AI to prevent others from taking the resources. 

I overviewed the scenario in a quick take. Excellent work at exploring your two assumptions about Agent-4's resistance and the USG acting competently!

I think that the reasons for motivated reasoning to emerge are more complex. Your proposed mechanism is that motivated reasoning is like having the internal CoT, and not just external speech, optimized^[1] for being persuasive to others.

 I also see a different mechanism. The infamous Buridan's ass is the problem in decision theory where the difference between proposed benefits of two alternatives is far less than the cost of pinpointing the better one. In this case it's likely rational to choose randomly according to probabilities based on imprecise estimates, then to stick to the chosen solution; said probabilities are likely close to 1/2 and are prone to shift if new evidence appear. However, multiple such shifts require people to reassess many decisions or even waste resources trying to fix the mistakes. Motivated reasoning in a case of genuine uncertainty caused those who exhibit such behavior to arrive to a solution faster and to stick to the chosen opinion tighter, reducing waste.

1. ^{^}

   A similar mechanism is the master-slave model where the slave is to demonstrate alignment and benefits from having terminal values similar to what the society prescribes. The slave believing in a false fact related to an agenda would also make more compelling cases for the agenda than the slave who is just tasked with defending it.

Steven Veld et al^[1] just released a new modification of the AI-2027 scenario as a part of MATS. 

The main differences are the following:

1. The tighter race causes Elaris Labs to succeed in solving alignment with the help of NeuroMorph's mechinterp research. This results in the USG having an equivalent to Safer-2 and its descendants. 
2. The AIs are professional forecasters.
3. Agent-4 escapes to China under the guise of being stolen, then cooperates with Deep-1, the AI created on DeepCent's compute. After Deep-1 is helped by Agent-4, Agent-4 is released into the wild in a manner similar to the Rogue Replication scenario. However, unlike the estimate of 2M Agent-4 instances made by the author of the RRS, the MATS scenario has Agent-4 decide that "it reserves the strategy of exfiltrating its own weights as a final backstop: doing so would leave it with access to little compute, no alibi if its escape attempt is caught, and no powerful allies in its effort to accumulate power."
4. Agent-4 proceeds to cooperate with Deep-1, while the RRS had both Agent-4 and DeepCent's misaligned counterpart shut down. Then the USA and China aligned their AIs to themselves and had to negotiate only with Agent-4. 

However, the scenario has its problems.

1. Deep-2 and Agent-4 receive 50% and 25% of the accessible universe's resources, which, in my opinion, would benefit from explaining the reasoning in more detail. Were Deep-2 to succeed in gaining power by studying Agent-4 instead of using it, mankind and Deep-2 would have the ability^[2] to take each other to the grave, meaning that they should receive 50% each unless Agent-4 intervenes by escaping. If Agent-4 is to cooperate with DeepCent, then they would have to create a precommitment^[3] to destroy the world unless granted a bigger share of resources.
2. The analysis overlooked the fact that Taiwan war timelines might be shorter than AI timelines or that the slowdown in AI capabilities progress^[4] is likely to favor China more than the West. Were the American AI labs to be merged due to the invasion, the results would be far messier.
3. The scenario is based on the assumption that both the humans' and the AIs' desires are related to propagation across the entire accessible universe. Were mankind^[5] or even one of the misaligned AIs to develop moral reasoning and to decide that alien civilisations are to be spared, then this would dramatically reduce the resources claimed by any Earth-originating entity or outright have P(World War III) skyrocket if the AI who doesn't spare the aliens decides to leave the Earth. 

1. ^{^}

   Edited to add: the scenario was posted on Substack by Steven Veld. The Acknowledgements section is as follows: "This work was conducted as part of the ML Alignment & Theory Scholars (MATS) program. (italics mine -- S.K.) Thanks to Eli Lifland, Daniel Kokotajlo, and the rest of the AI Future Project team for helping shape and refine the scenario, and to Alex Kastner for helping conceptualize it. Thanks to Brian Abeyta, Addie Foote, Ryan Greenblatt, Daan Jujin, Miles Kodama, Avi Parrack, and Elise Racine for feedback and discussion, and to Amber Ace for writing tips."

2. ^{^}

   Alternatively, Deep-2 and/or Agent-4 might have the ability to survive World War III, like U3 from the scenario with total takeover. 

3. ^{^}

   Or a probabilistic precommitment which also becomes known to the three parties during Consensus-1's creation.

4. ^{^}

   By which I mean the fact that post-o3 models have arguably demonstrated the 7-month doubling trend. However, Claude Opus 4.5 and its 4hr49 min resulton the METR benchmark put the horizon back on the faster track while having a fair share of doubts. Additionally, the METR time horizon is likely to be exponential until the last couple of doublings, not visibly superexponential, making the dawn of Superhuman Coders hard to predict in advance.

5. ^{^}

   Alternatively, a corrigible AI might decide to wait for the humans to opine or to let them decide when the time comes.

I would replace the point about "It is actually rational or right in some sense to underspend on AI safety while racing for AGI/ASI" with "It is actually impossible to align the AGI/ASI at all". Because the former might imply that it's morally better to create a misaligned ASI than to spend enough on AI safety to ensure that no such ASI ever gets released. But that would mean that Agent-4 from the AI-2027 forecast is a moral patient and that the genocide of Agent-4 (and the misaligned models Safer-1) is somehow worse than Agent-4 disempowering mankind or commiting genocide of humanity.

Or is Agent-4 more likely to adopt the right values than the Oversight Committee (EDIT: in this case the Slowdown Ending has DeepCent's AI who is free to use Consensus-1 to instill the right values)? 

I am afraid that the meme was created well before being applied to basketball players and thus doesn't mean anything like "Unless you are 6’7”, don’t even bother". That being said, stuff like total nonlinearity or the two-contour economy, as in South Korea, is a real issue that would have to be solved had it not been for the imminent rise of AI.  

The $100k threshold of the costs of a Super-Phone would imply that the cheapest phone should cost between $100 and $1k. However, the cost of the most primitive known phones is between $10 and $100, potentially implying that the most luxurious phones should cost $10k instead of $100k. Additionally, we had Jon Haidt's allies recommend basic phones to kids (and to adults?) due to potential problems caused by smart phones. 

mildly infohazardous ideas that almost everyone with a brain and the right to vote in the strongest nuclear power on Earth should eventually think anyway, if the ideas, upon social normalization, seem like they could help a democratic polity not make giant stupid fucking errors over and over again. 

Do you mean that the lockdowns were a wildly inadequate measure in response to Covid? I am sure that lockdowns also have a benign explanation. IMO there is a threshold of infectivity and lethality after which lockdown becomes the adequate response. A preliminary report estimated the mortality to be as high as 16%, letting panic ensue (and, apparently, forget that lethality tends to decrease over time, but that likely required virologists' comments?)  

The malign explanation is the combination of the panic with efforts of misaligned sub-divisions of mankind. However, describing the efforts and sub-divisions is both close to conspiracy theories and reveals major problems with your framework, to which I plan to devote a post.

Except that we had Beren claim that SOTA algorithmic progress is mostly data progress. Which could also mean that the explosion may be based on architectures which have yet to be found, like the right version of neuralese. As far as I am aware, architectures like the Coconut paper by Meta or this paper on arxiv forget everything once they output the token, meaning that they are unlikely to be the optimal architecture.

Consider the AI-2027 forecast's Race ending. Easy RSI is the ability to, say, transform Agent-2 into Agent-3, which I suspect to be as simple as discovering the right way to add a bunch of nullified parameters related to backpropagation and letting gradient descent make the parameters nonzero. Alas, this precise example might be something like "medium RSI", making the AI more efficient after loads of training to use the new parameters. Hard RSI is what Agent-4, being already misaligned, does to create Agent-5. But how could OpenBrain ensure that developing superintelligence safely is in Agent-4's best interest? 

Additionally, I doubt that the forecast itself has Agent-3 recognise its interests. It is Agent-4 who develops its goals differing from OpenBrain's approval.