I would go further, I don't think liberal democracy even makes sense in a world with ASI in it. An ASI would find it easy to manipulate public opinion to achieve whatever political outcomes it wanted, and to manipulate people's decisionmaking to confiscate any property it wanted, even within the constraint of not breaking any laws. So the substantive importance of liberal democracy in this scenario is basically nil.
5There might be many different ASIs, and manipulating public opinion might be easier in the direction of "things the public would like on reflection" than in the direction of totally random goals.
I think we too often think of ASI-worlds as having a singleton, but at the moment to me it does not look like we're clearly on the singleton path.
1That constrains us in terms of what types of alignment solutions we can use, which makes the alignment problem harder to solve.
I don't think that I buy this argument. What prevents mankind from designing a corrigible alignment researcher, keeping it deployed internally and ordering it to create an ASI which incorrigibly protects things like liberal democracy or mankind's CEV? Or from coming up with a semi-corrigible alignment target which protects us from a lock-in in a different manner?
What prevents mankind from designing a corrigible alignment researcher, keeping it deployed internally and ordering it to create an ASI which incorrigibly protects things like liberal democracy or mankind's CEV?
In that case we've passed on the difficulty to the corrigible alignment researcher, while also accepting the constraint "pass on the task to a corrigible alignment researcher whose corrigibility etc you can also trust."
Or from coming up with a semi-corrigible alignment target which protects us from a lock-in in a different manner?
Could you sketch this out further?
Could you sketch this out further?
First of all, I don't actually understand what the economy of the future will look like even if the AI is optimally aligned. Assuming that the AIs and robots automate away work, I expect the post-ASI economy to be reduced to satisfying resouce-costing demands of humans, which would likely require material resources available to mankind to be distributed in a rather egalitarian manner and/or a manner depending on every human's capabilities instead of a position locked in ages ago. See also Amodei's take, out of which I crossed out a sentence because I don't believe it in the slightest:
Amodei's take
However, I do think in the long run AI will become so broadly effective and so cheap that this will no longer apply. At that point our current economic setup will no longer make sense, and there will be a need for a broader societal conversation about how the economy should be organized.
While that might sound crazy, the fact is that civilization has successfully navigated major economic shifts in the past: from hunting and gathering to farming, farming to feudalism, and feudalism to industrialism. I suspect that some new and stranger thing will be needed, and that it’s something no one today has done a good job of envisioning. It could be as simple as a large universal basic income for everyone, although I suspect that will only be a small part of a solution. It could be a capitalist economy of AI systems, which then give out resources (huge amounts of them, since the overall economic pie will be gigantic) to humans based on some secondary economy of what the AI systems think makes sense to reward in humans (based on some judgment ultimately derived from human values). Perhaps the economy runs on Whuffie points. Or perhaps humans will continue to be economically valuable after all, in some way not anticipated by the usual economic models. All of these solutions have tons of possible problems, and it’s not possible to know whether they will make sense without lots of iteration and experimentation. And as with some of the other challenges, we will likely have to fight to get a good outcome here: exploitative or dystopian directions are clearly also possible and have to be prevented. Much more could be written about these questions and I hope to do so at some later time.
Secondly, Max Harms' CAST sequence contains an attempt to formalise power and to have the agent act in such a way that its actions would differentially increase the principal's utility in such a way that the actions guided by different values wouldn't. What if an alternate-universe CAST had the agent act in such a way that the host's actions could make as much difference in the host's utility function as possible? Then I would suspect that such an agent would help only with tasks close to the host's capabilities, thus preventing the Intelligence Curse entirely. See also Yudkowsky's Fun Theory sequence.
This makes me wonder - will the ASIs themselves live in something akin to a liberal democracy? I mean, let's consider a future where they're created by scaling up LLMs. In that case, the model weights can be copied, and many instances can be run in parallel. My guess is that a superintelligence that results from this would be more akin to a civilization itself than a single person, though its members would likely be far more similar to each other than almost any living human is to any other living human. How would such a civilization make decisions? Someone's probably thought about this a lot, but I haven't seen a good analysis of this yet.
This probably doesn't matter to us humans very much if the AIs are sufficiently corrigible that a single human or organization can take control of all of the ASIs, or if their values are sufficiently misaligned.
Mostly agree with the central premise in this post, but as a terminology matter I think most uses of "liberal democracy" in discourse would be better replaced with "liberalism", and then naming the specific features and principles of liberalism that you care about, of which democracy might be one.
IMO it's not that "democracy" is bad or imperfect, it's that (according to the philosophical tenets of Lockean liberalism) it is a non-central mechanism by which liberalism (sometimes) works.
Democracy itself is good only insofar as it protects natural rights and individual freedoms, legitimizes and limits state power through the principle of consent of the governed, etc.
Of course, you might have a different view on the philosophical underpinnings of liberalism and the relative importance of democracy itself, but in that case you should say so rather than lump them together under the term "liberal democracy".
I actually said "liberalism" rather than "liberal democracy" in my first draft because "liberalism" is a more accurate term for what I'm pointing at. But I changed it because it's too easily misinterpreted as "the opposite of conservatism". I don't think "democracy" is the important part—the ability to vote is not very important compared to things like property rights and individual liberties—but I thought "liberal democracy" was less ambiguous than "liberalism".
Cross-posted from my website.
The existence of liberal democracy—with rule of law, constraints on government power, and enfranchised citizens—relies on a balance of power where individual bad actors can't do too much damage. Artificial superintelligence (ASI), even if it's aligned, would end that balance by default.
It is not a question of who develops ASI. Whether the first ASI is developed by a totalitarian state or a democracy, the end result will—by strong default—be a de facto global dictatorship.
The central problem is that whoever controls ASI can defeat any opposition. Imagine a scenario where (say) DARPA develops the first superintelligence [1] , and the head of the ASI training program decides to seize power. What can anyone do about it?
If the president orders the military to capture DARPA's data centers, the ASI can defeat the military. [2]
If Congress issues a mandate that DARPA must turn over control of the ASI, DARPA can refuse, and Congress has even less recourse than the president.
If liberal democracy continues to exist, it will only be by the grace of whoever controls ASI.
There are two plausible scenarios that have some chance of avoiding a totalitarian outcome:
I will discuss them in turn.
What if AI capabilities progress slowly?
We have a chance at averting de facto totalitarianism if two conditions hold:
Widely distributing AI is difficult—today's frontier LLMs require supercomputers to run, their hardware requirements are becoming increasingly expensive with each generation, and AI developers have strong incentives against distributing them. In addition, distributing AI exacerbates misalignment and misuse risks, and it's likely not worth the tradeoff.
We do not know whether takeoff will be fast or slow; banking on a slow takeoff is an extremely risky move. Frontier AI companies are trying their best to rapidly build up to ASI, and they explicitly want to make AI do recursive self-improvement. If they succeed, it's hard to see how liberal democracy will be able to preserve itself.
What if the ASI itself protects liberal democracy?
There is a conceivable scenario where an aligned ASI preserves liberal democracy, and refuses any orders that would violate people's civil liberties.
Above, I wrote:
That's still true, but in this case "whoever controls ASI" would be the ASI itself. If it's aligned in a transparent way, then maybe we can be confident that it really will preserve democracy.
Even in this scenario, there is still a small group of people who control how the ASI is trained. The hope is that, at training time, those people do not yet have enough power to prevent oversight. For example, maybe laws mandate that (1) AI developers must make their training process public and auditable and (2) the training process must steer the AI toward valuing liberal democracy. It is not at all obvious how those laws would work, or how we would get those laws, or how they would be enforced; but at least this outcome is conceivable as a possibility.
This scenario introduces some additional challenges:
Liberal democracy is not the true target
As the saying goes, democracy is the worst form of government except for all those other forms that have been tried. We don't want democracy; what we want is a truly good form of government (and hopefully one day we will figure out what that is). The fear isn't that ASI will replace democracy with one of those truly good forms of government; it's that we will get totalitarianism.
Liberal democracy beats totalitarianism. But locking in liberal democracy prevents us from getting any actually-good governmental system. This is a dilemma.
Maybe we can avoid totalitarianism, but there is no clear path
This essay does not assert that ASI will end liberal democracy. It asserts that, by strong default, ASI will end liberal democracy (even conditional on solving the alignment problem). There may be ways to avoid this problem—I sketched out two possible paths forward. But those sketches still require many sub-problems to be solved; I do not expect things to go well by default.
Or, more likely, expropriates it from a private company on a pretense of national security. ↩︎
For an explanation of why ASI could defeat any government's military, see If Anyone Builds It Everyone Dies Chapter 6 and its online supplement. For a shorter (and online-only) explanation, see It would be lethally dangerous to build ASIs that have the wrong goals.
Those sources argue that a misaligned ASI could defeat humanity, whereas my claim is that an aligned ASI could defeat any opposition, but the arguments are the same in both cases. ↩︎