This is a linkpost for https://www.ai-2040.com/
New Comment
Mr Kokotajlo, one of the authors of this AI 2040 scenario, describes it as follows:
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I'm tempted to say, "will probably discover") and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I'm not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
...I’m not sure that actual existing efforts towards delaying AGI are helping. I think the path from here to AGI is bottlenecked by researchers playing with toy models, and publishing stuff on arXiv and GitHub. … Once the new paradigm is known and developed, the actors able to train ASI from scratch will probably number in the tens of thousands, spread all around the world. We’
1
1
1Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
Plan A largely doesn't agree with this. Since they expect the slowdown treaty to eventually fall apart--at which point we revert to the status quo arms race, which we all agree is bad--they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Scott Alexander had a nice little graphic on this point in his ACX post on Plan A:
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
11In my humble opinion, the AI 2040 document falls short of being such a plan because it does nothing to stop or slow research that might find (and I'm tempted to say, "will probably find") a fundamentally more efficient learning algorithm than what the leading labs are currently using.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
...In the past, companies have trained bigger and better AIs using both compute scaling (bigger training runs) and software progress (advances in AI algorithms—new paradigms, better training recipes, better data, etc.). Now, the Consortium tries to steer things so that the majority of improvement comes from increasing training compute.
Algorithms are information; it’s inherently difficult to stop them from proliferating, and the total research transparency means we aren’t even trying. [FN: There are a few nuances here. Some algorithmic progress is easily communicated (e.g., new architectures, better optimization algorithms), while other types cannot easily diffuse (e.g., huge libraries of RL environments, hardware-software codesign, scale or compute dependent algorithms). Regulations that the US and China a
8
Plan A does not advocate for what I consider the most potent brake on that "progress", namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
[...]
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind's freely giving away the knowledge described in the "Attention is all you need" paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the history of AI research, I know enough about the history of research and development in general to say with confidence that at least over the last 200 years, the vast majority of technological and scientific progress over all fields was caused by these acts of freely giving away insights. I'd be very surprised to learn that AI is an exception to that generality: the vast amount of potential revenue and profit to be made with AI makes giving-away less likely, but the idealism and (false) sense of an altruistic purpose makes giving-away more likely.
To slow down the rate of advance of a field, we would prevent these acts of giving-away from happening: if we could somehow prevent all insights discovered inside OpenAI from disseminating beyond OpenAI, that would IMHO be an improvement (though shutting OpenAI down and requiring all its employees to find something else to do other than AI research would be strictly better)!
5
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you're proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. ("Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa".) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
5
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
4
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others' knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others' tech but aren't publishing.
(My two parantheses were me speculating about possible answers to this question.)
2
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
4
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it's generally bad to restrict speech for the usual reasons. Lukas Finnveden's comments elsethread are good. I'm open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
5
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher's ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase "assessing AI risks and safety cases" that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. "We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don't like, we're going to apply strong pressure on them," is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that
3
That's reasonable. We put Plan S in there as an available option because we do think it's a serious proposal and might even be better than Plan A.
2
I've been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I've stopped now.
AI 2040 has given me things to think about.
4
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain's plasticity and it wouldn't matter if it doesn't also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.
“I wish it need not have happened in my time,” said Frodo. “So do I,” said Gandalf, “and so do all who live to see such times.”
Given the level of political will and international coordination in the story, why can't they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made - something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn't have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I'm not sure the financial loss would be more than 2x bigger ...
1If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn't really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It's 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it's basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they'll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don't think it's obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India's leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it's smarter than other models, even though it doesn't directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transpare...
8
I think I'm just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it's likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don't see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it's over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It's also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can't source new GPUs, no? I don't remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that's not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that's bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn't have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the co
2
The game theory doesn't work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
2
I think countries sometimes have honor and follow their commitments even if it's not locally game theoretically optimal.
2
Sure, but it's worth noting that's a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.
6
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
4
I think it's pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn't restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don't believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
2
See https://ai-2040.com/supplements/deal-decline for my probabilistic estimates here.
4
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there's probably other ways) is if you think that most of the value of "~infinity growth now" would already have been lost by moving to "~infinity growth in 10 years". But that doesn't seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than "singularity in 10 years" by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
2
Yes, that's fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won't really be lifted or progress will be reversed in other ways. (E.g. I don't expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge - I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
2
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there's much less hope of that if there's an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn't that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don't currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they'll have that big of an effect?
It's not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn't explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
- What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn't Significantly Enhancing Adult Intelligence With Gene Editing Possible?
- What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
- The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we'll have a powerful AI capable of generating better ideas): "If we tr
kman here. I should mention that I now think that post was very overly optimistic, and I don't think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn't seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as "scenario S".
(I still think adult enhancement is worth pursuing as well. I'm currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it'll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
8
1. (See also kman's comment.) Besides the technical difficulties with trying the experiments, it's quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on "crystallized intelligence"; to go bigger, I'm guessing you want more child-like learning capabilities; though that's just speculation.) Even for IQ genes that are expressed in adult brains, it's plausible their effects on IQ are mainly or linearly coming from childhood effects.
2. My uninformed impression of E11 is that it's very cool, but also is not the bottleneck to WBE or similar. The connectome isn't the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
3. I don't believe the claim about simulating something close enough to neurons; see the previous point. Re/ BCIs, see also: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=ff9c8PHtuFKfwE6Yj
[...]
I highly doubt this is relevant.
In general there is the issue that it's hard to test what you're interested in; and you're not constrained to the natural manifold (as with reprogenetics), so there's little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
8
I agree that BCIs probably wouldn't do that much. (Though:) There are a few threads that one can imagine, such as:
* networking people together and having them spend a bunch of time learning how to use that;
* prosthetically increased long-range connectivity;
* some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
* networking with some sort of neural organoids to get "more cortex".
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I'm not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like "do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes". It's hard to tell and I think it's unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
If that's a crux, happy to give more info.
4
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
2
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the "you're a everyday citizen" scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)
I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don't want to disrupt that project, but after the AI timelines update comes up, I'd like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue's substack post on A Stargate for Data.
Especially in worlds where we don't get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
13
Trying to understand your model:
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today's AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By "software intelligence explosion", do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
2
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we're gonna work on it! It's fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We've thought about this in the past and decided that it probably wasn't worth including in our model. But we should think about it more.
Total research transparency feels a bit too galaxy-brained for me. It makes non-robust assumptions that newly discovered techniques won't be usable to enhance already existing open-weights models to excessively dangerous capability levels. I also think the disincentive for research is overstated as it neglects first-mover advantage.
Two brief random thoughts:
-
I think it's great how large a role robotics plays in this analysis. In general, I feel like robotics isn't given nearly the amount of attention it deserves. Without robotics, AI's influence on the world is bottlenecked by human hands. So the speed of the feedback loop between AI building better robots to build better robot factories to build better robots seems like a very important crux. So bravo for taking that issue seriously.
-
Regarding the proposal to make AI research public but not model weights, I think it might be worth giving a bit more thought to the idea the whatever is made public essentially becomes a public good, and public goods tend to be under-supplied—or in this case perhaps "under-supplied", as reducing the amount of AI research supplied might be considered a good thing or perhaps even the primary benefit of such a policy. Theoretically, if people really were forced to make all AI research public in a way that was instantly useful to competitors, it seems like that would either drastically reduce investment in research or force people to invest in research that other people couldn't take advantage of, e.g. research that is only usef
3
Yep as we say in the piece, we think that it's better for AI progress to happen mostly via scaling compute rather than mostly via algorithms (that is, in the context of a Plan-A style reversible deal with mutually assured chip destruction). So we think it's a feature, not a bug, that publishing the algos disincentivizes investment in algo research. There'll still be some algo progress in Plan A and that's fine, we model it in the model. (Indeed, it's probably impossible to solve alignment without making some algo progress as a side-effect)
Someone made a comic version of this that may be more digestible for some audiences.
So... we make deals with the misaligned AI, to reward them for cooperation... but not the aligned AI?
I think the idea is an aligned AI would want what we want, so 'rewarding' it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we'll be less likely to shut it down but still respect the 2% diff.)
4
It was only a brief expandable, but yes, we'd want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
1
i'm sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don't want nitpick it too hard
i just worry there's a strong chance that "give affordances to aligned AI" will end up combining with "aligned AI are the ones who think it's dangerous for AI to want affordances" in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn't want to scare off bewildered politicians who might be reading, so you're listing the most prosocial and convenient possibilities. nobody said the scenario wasn't optimistic, after all
Overall really glad this was made! Wanted to flag that there appears to be a UI bug on the side menu of the supplemental pages where the menu overlaps the text.
3
I can't reproduce this, what browser and OS is this?
2
Google Chrome and Mac OS.
Strangely I can't reproduce it either. It's happening in one of the tabs I have pulled up but not the others. I used to do web development and I checked the obvious things. Not sure whats going on but the other tabs look correct.
2
Weird. Well, I'll watch out for it.
This looks very interesting, thanks for writing!
One minor note for clarity: I initially interpreted the term "employment rate" to be the complement of the unemployment rate and was confused why it was so high (and I didn't realize it was clickable). For example in 2027, its at 62% with only a 1.2x AI R&D speedup. (In this picture, I selected 2027 and hovered over employment to get the tooltip)
I didn't realize that there was an official meaning of the term (which you had meant) like OECD's official definition.
But I think many readers are likely to misi...
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained wh...
1I'm at 2029 or 2028 median now, not sure. We'll try to do a reassessment of timelines soon and get out an update.
71I tried to read Plan A, I'll probably try again, but I find it hard to take seriously a scenario in which there are millions of human-level AIs, and in which all algorithmic progress is public, but the human race still has a choice about whether to hand over power, after ten years of that... The whole thing reads like the kind of SF in which superintelligent takeover is artificially delayed so there can be lots of human-level plot twists.
6
I guess you disagree about some combination of:
(a) Under conditions of transparency and slowdown, with companies required to make solid safety cases etc., control-based safety cases will be made strong enough to work up until about top human expert level AI
(b) If so, alignment will be adequately solved within a couple of years?
If b is false then we'd have a similar scenario but stretched out over more years.
If a is false then we'd need to pause at a lower level of capability than top expert level, whichever level is low enough that control works.
4
If you want control that works, I think you need to turn the clock back, before chain of thought and reasoning models. See my comment to @Vladimir_Nesov. Vladimir has a nuanced defense of his position, but I think OpenAI's o-series (i.e. GPT-5) may have been the point of no return.
It would be nice to link a supplement with some actions that people with different levels of resources can take to make progress on this. I’m sure this is coming, hopefully soon for momentum purposes.
We have a get involved page for verification in particular. It tracks the current state of play wrt verification and what still needs to be done: https://ai-2040.com/supplements/verification-plan/get-involved
We might do a blog post on other actions as well.
I really liked it; I think it was very well-written. I have a quick response to the epilogue: https://www.lesswrong.com/posts/EANs7YerYXmaXF9FE/don-t-normalize-a-permanent-underclass-even-a-rich-one-1. But, overall, thank you so much to the AI Futures Project for all of the hard work that went into this!
Hi! I took some notes as I read. I'm sure some of them are addressed in what I didn't read (I only read Plan A, no supplements), or I didn't pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
- This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
- Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully or accidentally leak it to other parties?
- Could the US government just decide not to give a dividend? Is there some other use it would rather put the money to? Could it, if it wanted to? Would the public trust blind-verified sousveillance?
- If the dangers of AI persuasion are countered by having your own AI, this seems to only move the battlefield up one layer - now AIs are trying to persuade other AIs. Do we know if the asymmetries favor defense here?
- Is a basin of sanity a coherent or evidenced concept? I know that in the rationalist sphere we like to think we've gotten the path to sanity down, but, is it guaranteed that we're
A key part of this strategy is deterrence, "Mutually Assured Compute Destruction" which gets its own section. It doesn't mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1/3 of the MAIM discussion on verification and how to do this in a multilateral way.
Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It's a bewildering unforced error--it's extremely related, it's a certainly a nontrivial idea, and I told you all this recently--I hope you all fix it.
6
Sorry that was an oversight, we'll edit to include a footnote citing MAIM.
I'm excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We'll try to monitor it and respond to many of the comments.
This is really thought provoking work. I think the power of personal diplomacy may have been overplayed at the point where 'the President' and Xi are mentioned hashing out capabilities limitations over phone call. I understand it's an optimistic scenario, but these things are generally procedural. Is that a baked in assumption?-- that the standard means of accomplishing international cooperation will have to fall by the wayside in favor of a more gung-ho, personal style of diplomacy if something like The Coalition is to be achieved? I'm generally interested in how you guys made your assumptions re: where incentive structures could carry the day vs. where more concerted political will would have to be built.
I'm very impressed by the proposed Total Research Transparency. I actually found it appealing even beyond the many reasons mentioned in the plan. It takes advantage of key properties of the current training paradigm, and this is actually desirable, because these properties are likely to remain in future AI systems:
- Model training and serving will keep incentivizing a small number of large neural networks, with agent diversity coming from context. Efficiency of batched inference is too large, the learning algorithm and the hardware have co-evolved around it. My opinion is that continual learning will be hard on hardware like the current one because of loss in batching efficiency. And radically different hardware will take longer to develop.
- The auto-regressive architecture will also likely remain, because having a chain of tokens empowers the model with the ability to perform inherently serial computation, making it strictly more computationally powerful. This makes it possible to create verification schemes with teeth (via CoT monitoring etc).
So open sourcing the algorithms, closing the weights and monitoring inference seems like the right balance.
If we create an actual recommendatio...
For the past months, I've had many sleepless nights thinking over a scenario which I couldn't resolve completely, a scenario that involves a "sovereign leap" by smaller countries in their last moments to combat permanent disempowerment. They deduce that the best means of balancing the asymmetry of superintelligence is creating weapons of biosphere destruction.
They precommit to destroying the biosphere if the US and China try to permanently lock in the status quo in their ASI deal, or destroy or poke their biological weapons in any way. There would be most...
Note #185 is misleadingly worded. It sounds like the probes could reach the entire reachable universe in 6 hours, whereas the cited paper says 6 hours of Dyson-sphere output could power their launch.
I have read the entire post and sub-scenarios and know I am not able to gauge the reality and speed of what is said to lie before us. Nonetheless, I am in shock.
Another Tolkien quote ""It does not do to leave a live dragon out of your calculations, if you live near him."
And so we do.
Questions/thoughts:
- What will be the cost of a bag of potato chips if everyone is making $10MM?
- What the heck is all the work being done?
- What is the cost to the environment with such large GDP increases?
- Will anyone have children?
- What happens to all the parchment books squirreled away in dusty libraries? Do they get scanned and their information added to the training data?
- Simulating past histories seems not far from Nick Bostrom's simulation trilemma scenario.
The notion of radical transparency and truth fascinates me. Imagine:
- MoTru. The Museum of Truth implemented by neutral curAItors, AIrchitects and desAIners.
- All Sides or Less Wrong becomes the AI-driven trusted global news network.
- Public and private sides of life, politics, and culture dissolve. Everyone knows everything about everything, or could, and an the Machines of Loving Grace era is a result.
I lean toward think a bad outcome will with bad ac...
11
I concur and I would also add - where are people living? Space stations and terraformed planets are mentioned but if hyperrealistic virtual reality is possible, wouldn't living there be more efficient?
For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!
It’s called AI 2040: Plan A.
It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.
It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.
As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.
Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer.
What’s next for us?
Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that AI 2040: Plan A sparks. Beyond that, we aren’t sure yet. In general our mission is to help make AGI go well, and now we’ve tried out both forecasting and planning. Maybe we’ll get started on another big scenario. On the other hand, these megaprojects take so much time…