Instead, it’s the point of no return—the day we AI risk reducers lose the ability to significantly reduce AI risk. This might happen years before classic milestones like “World GWP doubles in four years” and “Superhuman AGI is deployed."
The rest of this post explains, justifies, and expands on this obvious but underappreciated idea. (Toby Ord appreciates it; see quote below). I found myself explaining it repeatedly, so I wrote this post as a reference.
AI timelines often come up in career planning conversations. Insofar as AI timelines are short, career plans which take a long time to pay off are a bad idea, because by the time you reap the benefits of the plans it may already be too late. It may already be too late because AI takeover may already have happened.
But this isn’t quite right, at least not when “AI takeover” is interpreted in the obvious way, as meaning that an AI or group of AIs is firmly in political control of the world, ordering humans about, monopolizing violence, etc. Even if AIs don’t yet have that sort of political control, it may already be too late. Here are three examples: [UPDATE: More fleshed-out examples can be found in this new post.]
-
Superhuman agent AGI is still in its box but nobody knows how to align it and other actors are going to make their own version soon, and there isn’t enough time to convince them of the risks. They will make and deploy agent AGI, it will be unaligned, and we have no way to oppose it except with our own unaligned AGI. Even if it takes years to actually conquer the world, it’s already game over.
-
Various weak and narrow AIs are embedded in the economy and beginning to drive a slow takeoff; capabilities are improving much faster than safety/alignment techniques and due to all the money being made there’s too much political opposition to slowing down capability growth or keeping AIs out of positions of power. We wish we had done more safety/alignment research earlier, or built a political movement earlier when opposition was lower.
-
Persuasion tools have destroyed collective epistemology in the relevant places. AI isn’t very capable yet, except in the narrow domain of persuasion, but everything has become so politicized and tribal that we have no hope of getting AI projects or governments to take AI risk seriously. Their attention is dominated by the topics and ideas of powerful ideological factions that have access to more money and data (and thus better persuasion tools) than us. Alternatively, maybe we ourselves have fallen apart as a community, or become less good at seeking the truth and finding high-impact plans.
Conclusion: We should remember that when trying to predict the date of AI takeover, what we care about is the date it’s too late for us to change the direction things are going; the date we have significantly less influence over the course of the future than we used to; the point of no return.
This is basically what Toby Ord said about x-risk: “So either because we’ve gone extinct or because there’s been some kind of irrevocable collapse of civilization or something similar. Or, in the case of climate change, where the effects are very delayed that we’re past the point of no return or something like that. So the idea is that we should focus on the time of action and the time when you can do something about it rather than the time when the particular event happens.”
Of course, influence over the future might not disappear all on one day; maybe there’ll be a gradual loss of control over several years. For that matter, maybe this gradual loss of control began years ago and continues now... We should keep these possibilities in mind as well.
[Edit: I now realize that I should distinguish between AI-induced points of no return and other points of no return. Our timelines forecasts and takeoff speeds discussions are talking about AI, so we should interpret them as being about AI-induced points of no return. Our all-things-considered view on e.g. whether to go to grad school should be informed by AI-induced-PONR timelines and also "timelines" for things like nuclear war, pandemics, etc.]
If AI is un-alignable (or at least significantly easier to create than to prevent dis-alignment), the point of no return was 1837. If Babbage had kept his mouth shut, maybe we could have avoided this path.
But really, it's a mistake to think of it as a single point in time. There's a slew of contributing factors, happening over a long time period. It's somewhat similar to recent discussions about human revolutions (https://www.lesswrong.com/posts/osYFcQtxnRKB4F4HA/a-tale-from-communist-china and others). It happens slowly, then quickly, and the possible interventions are very unclear at any point.
This is true, even if AI takeover never happens. The environment changes significantly over a human lifetime, and the only reasonable strategy is to thread a path that has BOTH long-term impact (to the extent that you can predict anything) AND short-term satisfaction. “Find a job you enjoy doing, and you will never have to work a day in your life.” remains solid advice, regardless of reasons for uncertainty.
I don't agree with your answer to your rhetorical question. A kitchen knife can cause injury and death pretty easily, but while it can be a weapon I wouldn't say that kitchen knives are "inherently a weapons technology". A brick can cause injury and death pretty easily too, and bricks are certainly not "inherently a weapons technology".
I would only say that something is "inherently a weapons technology" if (1) a major motivation for its development is (broadly speaking) military and/or (2) what it's best at is causing injury, destruction and death.
Military... (read more)