The AI 2040 framing is built around the belief in an approaching fire. I think there's currently no visibly approaching fire, LLMs in the current paradigm learn deep skills too slowly (where RLVR rather than in-context learning is necessary), but they do finance a massive buildup of dry tinder that makes the danger of a fire much higher than it was 10 years ago. They might even start a fire at some point, but not at a predictable time, and not much faster than some human researchers might start that fire, provided they do have access to all this dry tinder.
So it should all be about the dry tinder (and the tinder factories with their extremely complicated supply chains). Even the stopping of the fire might be a mirage, except the putative cause of that hypothetical fire (LLMs) is what's financing the dry tinder buildup, so addressing this cause might be beneficial in principle, even if it's not implicated in an approaching fire. Saying that it is implicated if in fact it isn't would be a big problem for a policy argument (especially at this level of disruption).
I think there's currently no visibly approaching fire
My understanding of your point-of-view is that LLM-based AI doesn't look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
your point-of-view ... LLM-based AI ... lacks continuous learning and sample efficiency
Sample efficiency is not a problem with RLVR, LLMs can reach or exceed world class human capability (at any given sufficiently specific thing) with a reasonable amount of training data in the form of tasks/environments/graders. And this will probably get automated (within the remaining runway of raw scaling for LLMs), with slow prosaic RSI of LLMs automatically preparing such training data and using it to train the next model. Continual learning in the sense of unlimited context is not helpful because in-context learning doesn't train deep skills (LLMs can't learn to play chess well by looking at memory notes in their context).
The crucial constraint is that retraining with RLVR in the current paradigm (by preparing the next model) happens too slowly, much slower than token generation. Amdahl's law then destroys the advantage of 100x faster idea generation and problem solving (using the current skills and understanding) in the overall task of research, because the subtask of learning novel deep skills doesn't have a speedup compared to humans (and in fact might happen much slower). Raw intelligence doesn't get around the lack of a speedup, because LLMs won't get to vastly superhuman levels within the remaining runway of scaling in the next few years.
So continual learning in the sense of doing something like RLVR on the fly would indeed solve this hobbling, but it's currently an unsolved problem. If LLMs can't do conceptual research faster than humans because of their long learning loop (even if the learning becomes automated in a general way, making LLMs able to eventually learn any specific skill), and they aren't vastly superhuman in coming up with ideas far outside their current skills and understanding, then they won't be crucial in solving this hobbling either.
in the current paradigm
Could you explain how, except for Plan A, mankind is to prevent anyone from creating the potentially hazardous techniques which are neuralese and continual learning? Was such a prevention the very point of Plan A?
So if the deal breaks down in 2033, the intelligence explosion happens 25x faster; in 2040 it happens ~600x faster. If the intelligence explosion would have lasted a year, it will now last just a couple of weeks or as little as a single day!
Surely not - compute is not the only input to progress, and some things need serial time. If you have a task where you have a 1 week gap between making an attempt and seeing how it went, then at week 0 you have only the data from before your first attempt, and no you're not getting the results for that first round of trials until after that week is up no matter how much compute you throw at it. Maybe you can route around most things which have a feedback loop, but compressing a year of progress into a day means you've found a way to route around basically all of the bottlenecks, immediately, with no time for your model of how to bypass that bottleneck to make contact with reality first.
A group is worried about an approaching fire spreading rapidly through their city. They manage to halt the fire outside the city gates. Meanwhile they build massive physical structures to help them study and guide the fire safely. But these structures are all made of highly flammable dry tinder! If they lose control of the fire, it will now rip through the city much more quickly.
This is a (flawed!1) analogy for Plan A, AIFP’s plan for how the world can safely develop superintelligence.
The key risk Plan A addresses is that of an uncontrolled software-driven intelligence explosion. Its remedy is a US-China deal to pause software progress at the brink of the intelligence explosion, while building up massive amounts of compute. This compute could be helpful for making AI safer! But it would also dramatically speed up an intelligence explosion if the deal breaks down. The cure risks making the disease much worse.
In particular, the deal pauses software progress from ~2030, but compute keeps scaling. By 2033 total compute has increased by ~100x and by 2040 by a further ~100x.2 AIFP estimates each 10x in compute speeds up the intelligence explosion by ~5x. So if the deal breaks down in 2033, the intelligence explosion happens 25x faster; in 2040 it happens ~600x faster. If the intelligence explosion would have lasted a year, it will now last just a couple of weeks or as little as a single day!
The dry tinder isn’t limited to the compute overhang. There is also an increasing overhang of software techniques. Companies can research new algorithms but must make them public — the Consortium then decides which algorithms are too dangerous to implement. A defector could ~instantaneously gain a big capabilities advantage by implementing all the banned software techniques.
There are two dimensions to dry tinder. We’ve discussed the first: the max speed of a software-driven intelligence explosion. The second is the number of distinct projects that could do a dangerously fast intelligence explosion. During the 2030s, more and more companies and countries amass the necessary compute and software for this.
So dry tinder creates two problems for Plan A:
Plan A’s solution
AIFP are well aware of the dry tinder problem.3 This is why Mutually Assured Compute Destruction (MACD) is so central to Plan A.
The US puts ~all their data centres in Mongolia and US chips require encrypted “continue” messages from China. If China detects that the US has broken the deal, they destroy the US data centres and cut off the “continue” messages.
And vice versa, China’s data centres are in Canada and require “continue” messages from the US.
The hope is that this commitment to MACD prevents a fast intelligence explosion despite all of the dry tinder lying around.
I see three challenges for MACD.
Challenge 1: quickly and reliably detecting violations
If detection takes multiple days, that could be too slow. The intelligence explosion may have finished. Or it may be midway through, and the defector exfiltrates the weights and algorithmic insights just before their data centres are destroyed,4 giving them a decisive headstart in the subsequent race to rebuild.
I’m not a verification expert, but getting high reliability in an adversarial setting like this seems very difficult, given possibilities like side channel attacks or compromising the verification infrastructure.
It seems especially tough if defectors can use millions of expert-human level AIs to search for vulnerabilities. So, as part of the plan, I propose that AIs worldwide have guardrails blocking this behaviour.5
Challenge 2: quickly destroying a defector’s compute
Again, a delay of days could be fatal. The defector could be poised to disrupt enemy military operations and airdrop troops to physically defend their data centres. And they could have secretly disabled any on-chip shutdown mechanisms.
Possible mitigation: the US can’t have any military presence near its data centres, or anywhere near the country where they’re located. (And likewise for other countries.)
The MACD dynamic also gets more complicated once there are >2 countries that could do a quick intelligence explosion. Let’s say Europe has caught up to the frontier. Where do their data centres go? Presumably, some within striking distance of the US, some within striking distance of China. But this means that the US can no longer unilaterally destroy Europe’s compute. It must trust China to do so. That means trusting its rival with its own national security — and worse, China and Europe could jointly stage an intelligence explosion that the US couldn’t stop.
Challenge 3: actually choosing to destroy the defector’s compute
I’ve discussed whether the Consortium has the technological capability to quickly implement MACD. But even with this capability, they may not choose to use it.
This is the challenge I’m most concerned about.
The analogy to nuclear MAD is not encouraging. The core logic of MAD is that the US doesn’t nuke Russia because it knows Russia would nuke it back. But the analogous equilibrium in MACD is that the US doesn’t destroy China’s data centres because it knows that China would then destroy US data centres. This is the opposite result from what Plan A needs!
If (say) the US defects, China will face a choice between:
The economic costs of MACD will be certain and enormous, and will immediately harm citizens and companies.
And MACD will not help China ultimately win the race by evening up the playing field. The US has already pulled ahead and can exfiltrate its weights and algorithms before its data centres are destroyed. (In fact, MACD would likely harm China’s chance of winning the race, because MACD reduces the fraction of global compute controlled by China.6)
So the stability of MACD rests on both US and China retaining a strong conviction that a fast intelligence explosion would pose unacceptable risks of misaligned AI takeover. But Presidents will change. Popular support for a pause may wane. Even conditional on the political will for Plan A existing initially, I worry that the MACD equilibrium will be too fragile. If a country is uncertain and deliberates for a few days, then that delay could be fatal.
What’s worse, the cost of MACD isn’t just losing your data centres. You have to destroy the fabs as well, otherwise the defecting country can quickly rebuild insane amounts of compute and do a super-fast intelligence explosion. And you have to destroy their new AI-powered industrial capacity too! Otherwise the billions of robots can quickly make loads of fabs which quickly make loads of data centres.
So Plan A has US fabs and robots confined to SEZs that are easily destroyable by China. And vice versa, China’s fabs and robots are easily destroyable by the US. In practice, this means the vast majority of the physical economy is destroyed in the case of MACD!
For a preview of the political economy here: NVIDIA successfully lobbied USG to remove export controls on its chips. The pressure against actually executing MACD — destroying most of the physical economy — would be orders of magnitude greater.
So even conditional on the political will for implementing Plan A existing, it currently seems unlikely that US/China follow through on MACD on fabs and robots (most of their economy!). And if they don’t, we are back to the problem of dry tinder and the dangerously fast intelligence explosion.
A possible fix to challenge #3: as the compute overhang grows, take the decision about whether to implement MACD out of human hands. Program highly reliable AI systems to bomb data centres if they detect a treaty violation, without needing human sign-off.
Is the alternative better?
It’s much easier to criticise than to improve, and so far this post has mostly done the former.
Plan A involves pausing software progress and scaling capabilities via hardware. The obvious alternative is to pause hardware scaling – ban compute and fab construction – and then slowly scale capabilities via software.
The key advantage of software scaling is that it removes unnecessary dry tinder, stabilizing the pause and reducing the dangers from a super-fast intelligence explosion.
The main drawback is that scaling capabilities via compute is likely safer than scaling capabilities via software. Eg you can run massive models with legible chains-of-thought rather than smaller models using neuralese. This is a big deal.
But we’ll need to figure out how to align neuralese-style systems eventually.7 So we have a choice:
Which is better depends on how stable the pause/slowdown is in the presence of dry tinder.
The other big drawback of software scaling is that software progress, unlike hardware progress, is likely to leak to “covert projects”. But this can be partially addressed by improved infosecurity, strongly favouring scale-dependent algorithms that can’t be used by projects with small amounts of compute, and co-specialising the hardware and software in legitimate projects so that the frontier software runs very inefficiently on the hardware of covert projects.
Overall, I’m not sure whether software-scaling or hardware-scaling is safer, and my sense is that AIFP isn’t sure either.
But I think there’s an argument for software-scaling based on option value:
Empowering China
Beyond dry tinder, I have one other worry about Plan A that I’d like to flag.
Today China is well behind the US in terms of compute and ability to push the algorithmic frontier. Plan A evens the scales on both fronts. China nearly catches up to the US on compute, and algorithms are all made public so they catch up algorithmically as well.
If the deal breaks down and a race begins without MACD (as I fear is likely), China is in a much stronger position to win because of Plan A.
Things are better if we implement MACD. In that case China returns to its previous compute disadvantage. But it’s still caught up to the algorithmic frontier and (most likely) the frontier of chip technology, a big advantage relative to today.
It’s very plausible that China ends up dominant in these scenarios, given its industrial advantage over the US.
I think this would be a pretty dire outcome. If China gets a decisive strategic advantage, that’s likely to result in a single global dictator – an absolute worst-case scenario from the perspective of extreme power concentration.
Again, this objection is far from decisive. Plan A reduces AI takeover risk much more than it increases the chance of a Chinese decisive strategic advantage. But I’m interested in variant plans that do more to maintain the current balance of AI power between the US and China.
Overall, I’m sympathetic to Plan A
The basic logic of the plan is sound: we will need to slow down AI progress at some point; this is a concrete plan for doing that.
Compute-scaling really does have significant alignment benefits. That might well outweigh the costs I’ve outlined here. But currently I lean towards starting with software scaling.
Footnotes
1 To patch the analogy: the fire must pass through the city eventually, so putting it out permanently isn’t an option; if the fire passes through safely it will make everyone amazingly rich; if a subgroup let the fire in and control it, they can become hugely powerful…
2 AIFP’s supplement states: “In the Plan A scenario, compute stock increases by roughly 0.7 OOMs/year from 2030 to 2033 and 0.1-0.3 OOMs/year from 2034 to 2040.” By the start of 2033 we’ve had three years of 0.7 OOMs/year compute growth, = ~2 OOMs. By 2040 we’ve had one further year of 0.7 OOMs/year growth, and 6 years of 0.2 OOMs/year growth, = ~2 OOMs more.
3 I raised it when giving feedback on a draft.
4 More specifically, the defector’s strategy would be: undermine the verification infrastructure; start a secret intelligence explosion; continually exfiltrate the weights and algorithmic insights; prepare to defend their data centres for as long as possible; prepare to destroy the opposing data centres as soon as their secret IE is detected.
5 Of course, this is tricky given that work to strengthen the verification will also involve studying its vulnerabilities.
6 In Plan A, the 2040 split of compute US / China / RoW is 35% / 20% / 45%. But countries also hold some compute in ‘cold storage’ that isn’t threatened by MACD. Cold storage compute matches countries’ pre-deal levels: 80% / 8% / 12%. So MACD dramatically reduces China’s compute relative to the US, from 1:2 to 1:10.
7 Lest we live with algorithmic dry tinder forever. I worry about the instability here, but perhaps aligned AI could enforce the ban indefinitely.