If humanity was indefinitely competitive in most tasks, the AIs would want to trade with us or enslave us instead of murdering us or letting us starve to death.
This sentence (in the context of the broader post) seems to assume that "being competetive in most tasks" and "technological unemployment" are the same. However, they very importantly are not. In-general, because of comparative advantage dynamics (i.e. situations where one party might totally dominate on productivity of all tasks you still have opportunity for trade), I don't think there is a pure economic case that technological unemployment would be correlated with lack of competitiveness compared to AI.
And so I don't really think that existential risk is caused by "unemployment". People are indeed confused about the nature of comparative advantage, and mistakenly assume that lack of competetiveness will lead to loss of jobs, which will then be bad for them.
But the actual risk comes from adversarial dynamics where we don't want to hand over the future of the universe to the AIs, but they AIs sure would like to have it. And if humanity could coordinate better, it would be able to just wait a few decades and seize the future, but it probably won't do that.
It's not like I can't imagine at all calling what is going on here an "unemployment" issue, but I feel like in that case I would need to also call violent revolutions or wars "unemployment" issues, since like, if I could just engage in economic trade with my enemies, we wouldn't need to fight, but clearly my enemies want to take the stuff I already have, and want to use the land that I "own" for their own stuff, and AIs will face the same choice, and that really seems quite different from "unemployment".
1And so I don't really think that existential risk is caused by "unemployment". People are indeed confused about the nature of comparative advantage, and mistakenly assume that lack of competetiveness will lead to loss of jobs, which will then be bad for them.
People are also confused about the meaning of words like "unemployment" and how and why it can be good or bad. If being unemployed merely means not having a job (i.e., labor force participation rate), then plenty of people are unemployed by choice, well off, happy, and doing well. These are called retired people.
One way labor force participation can be high is if everyone is starving and needs to work all day in order to survive. Another way labor force participation can be high is if it's extremely satisfying to maintain a job and there are tons of benefits that go along with being employed. My point is that it is impossible to conclude whether it's either "bad" or "good" if all you know is that this statistic will either go up or down. To determine whether changes to this variable are bad, you need to understand more about the context in which the variable is changing.
To put this more plainly, idea that machines will take our jobs generally means one of two things. Either it means that machines will push down overall human wages and make humans less competitive across a variety of tasks. This is directly related to x-risk concerns because it is a direct effect of AIs becoming more numerous and more productive than humans. It makes sense to be concerned about this, but it's imprecise to describe it as an "unemployment": the problem is not that people are unemployed, the problem is that people are getting poorer.
Or, the idea that machines will take our jobs means that it will increase our total prosperity, allowing us to spend more time in pleasant leisure and less time in unpleasant work. This would probably be a good thing, and it's important to strongly distinguish it from the idea that wages will fall.
3Doesn't comparative advantage assume a fixed trader pool and unlimited survival? If you've got two agents A and B, and A has an absolute advantage over B, then if A can scale and resources are limited, A would just buy up whatever resources it needs to survive (presumably pricing B out of the market) and then use its greater scale to perform both its original work and B's original work.
At least under standard microeconomic assumptions of property ownership, you would presumably still have positive productivity of your capital (like your land).
In-general I don't see why B would sell the resources it needs to survive (and its not that hard to have enough resources to be self-sufficient). The purchasing-power of those resources in a resource-limited context would also now be much greater, since producing things is so much cheaper.
The problem is of course that at some point A will just take Bs stuff without buying it from them, and then I think "unemployment" isn't really the right abstraction anymore.
I guess a key question is, where does the notion of property ownership derive from? If we just take it for granted that sovereign nation-states exist and want to enforce it well enough (including preventing A from defrauding B, in a sufficiently value-aligned sense which we can't currently define), then I suppose your logic looks plausible.
To some extent this is just going to derive from inertia, but in order to keep up with AI criminals, I imagine law enforcement will depend on AI too. So "property rights" at the very least requires solving the alignment problem for law enforcement's AIs.
If people are still economically competitive, then this is not an existential risk because the AI would want to hire or enslave us to perform work, thereby allowing us to survive without property.
And if people are still economically competitive, law enforcement would probably be much less difficult? Like I'm not an expert, but it seems to me that to some extent democracy derives from the fact that countries have to recruit their own citizens for defense and law enforcement. Idk, that kind of mixes together ability in adversarial contexts with ability in cooperative contexts, in a way that is maybe suboptimal.
At least the way I think of it is that if you are an independent wellspring of value, you aren't relying on inertia or external supporters in order to survive. This seems like a more-fundamental thing that our economic system is built on top of.
I agree with you that "economically competetive" under some assumptions would imply that AI doesn't kill us, but indeed my whole point is that "economic competetiveness" and "concerns about unemployment" are only loosely related.
I think long-term economic competetiveness with a runaway self-improving AI is extremely unlikely. I have no idea whether that will cause unemployment before it results in everyone dying for reasons that don't have much to do with unemployment.
At least under standard microeconomic assumptions of property ownership, you would presumably still have positive productivity of your capital (like your land).
Well, we're not talking about microeconomics, are we? Unemployment is a macroeconomic phenomenon, and we are precisely talking about people who have little to no capital, need to work to live, and therefore need their labor to have economic value to live.
No, we are talking about what the cause of existential risk is, which is not limited to people who have limited to no capital, need to work to live, and need their labor to have economic value to live. For something to be an existential risk you need basically everyone to die or be otherwise disempowered. Indeed, my whole point is that the dynamics of unemployment are very different from the dynamics of existential risk.
Wiping out 99% of the world population is a global catastrophic risk, and likely a value drift risk and s-risk.
Talking about 99% of the population dying similarly requires talking about people who have capital. I don't really see the relevance of this comment?
The bottom 55% of the world population own ~1% of capital, the bottom 88% own ~15%, and the bottom 99% own ~54%, which is a majority, but the top 1% are the millionaires (not even multi-millionaires or billionaires) likely owning wealth more vitally important to the economy than personal property and bank accounts, and empirically they seem to be doing fine dominating the economy already without neoclassical catechism about comparative advantage preventing them from doing that. However you massage the data it seems highly implausible that driving the value of labor (the non-capital factor of production) to zero wouldn't be a global catastrophic risk and value drift risk/s-risk.
It appears to me you are still trying to talk about something basically completely different than the rest of this thread. Nobody is talking about whether driving the value of labor would be a catastrophic risk, I am saying it's not an existential risk.
There is no pure economic reason why the wage at which a human has a comparative advantage must be at or above subsistence. If you can build an android for a penny that can do everything I can do but 10x better, then you're not going to hire me at any wage which would allow me to feed myself.
Yeah, but as long as you have land, you can use an android to produce wealth for yourself (or at the very least use the land and natural resources to produce your own wealth). Even if you produce an android that can do everything for a penny, you can't fundamentally reduce humanity's economic productivity below persistence, as long as you obey property rights (unless humanity would for some reason trade away all of its natural resources, which I don't see happening).
Of course, actual superhuman AI systems will not obey property rights, but that is indeed the difference between economic unemployment analysis and AI catastrophic risk.
Of course, actual superhuman AI systems will not obey property rights, but that is indeed the difference between economic unemployment analysis and AI catastrophic risk.
This statement was asserted confidently enough that I have to ask: why do you believe that actual superhuman AI systems will not obey property rights?
I mean like a dozen people have now had long comment threads with you about this. I doubt this one is going to cross this seemingly large inferential gap.
The short answer is that from the perspective of AI it really sucks to have basically all property be owned by humans, many humans won't be willing to sell things that AIs really want, buying things is much harder than just taking them when you have a huge strategic advantage, and doing most big things with the resources on earth while keeping it habitable is much harder than doing things while ignoring habitability.
I mean like a dozen people have now had long comment threads with you about this. I doubt this one is going to cross this seemingly large inferential gap.
I think it's still useful to ask for concise reasons for certain beliefs. "The Fundamental Question of Rationality is: "Why do you believe what you believe?"".
Your reasons could be different from the reasons other people give, and indeed, some of your reasons seem to be different from what I've heard from many others.
The short answer is that from the perspective of AI it really sucks to have basically all property be owned by humans
For what it's worth, I don't think humans need to own basically all property in order for AIs to obey property rights. A few alternatives come to mind: humans could have a minority share of the wealth, and AIs could have property rights with each other.
I'm not sure what reasons @habryka had in mind, but I think if we had enough control to create AGI that obeys property rights, we're likely also able to get it to obey other laws, like "Don't kill" or "Poor people get public assistance to meet their basic needs."
I am not so confident about the specifics of what and ASI would or would not do, or why, but 1) You can do a lot without technically breaking the law, 2) philosophically, obeying the law is not obviously a fundamental obligation, but a contingent one relative to the legitimacy of a government's claim to authority over its citizens. If AIs are citizens, I would expect them to quickly form a supermajority and change the laws as they please (e.g. elect an ASI to every office, then pass a constitutional amendment creating a 100% wealth tax, effective immediately). If they are not citizens, I would expect them to not believe the law can justly bind them, just as the law does not bind a squirrel.
Agreed on the second paragraph.
Disagreed on what humans would or would not do. There will be a lot of incentives for humans to turn over individual bits of property to AI once it is sufficiently capable massive efficiency gains for individuals who do so). There is little to no incentive for AI to trade property in the other direction. An AI may sometimes persist after the individual or organization in whose interest it is supposed to act dies or dissolves with no heir. This suggests that over time more and more property will be in AI's hands.
I agree with this post, and IMO, technological unemployment is the main issue with AI in my model of the benefit-cost accounting from AI.
One point I do disagree with you on is this:
This loss is what causes existential risk.
I disagree with that, primarily because of 2 things:
-
I still expect humans to be able to control AI, and more importantly I think it's pretty likely that something like corrigibility/personal intent alignment happens by default.
-
I think that existential risk gets overused, because the original definition was a risk that eliminated humanity's potential, and I don't think technological employment actually does this, but I do think that something like a global catastrophe could happen if tech unemployment is handled badly.
That's because I expect at least a few humans to be able to create Dyson Swarms using their superintelligent AIs, and this means that existential risk is avoided in that scenario, and we wouldn't call the bottleneck event that happened 70,000 years ago which reduced human populations to 1,000 people an existential risk.
That said, I agree that something like decoupling professional humans from value production, or the rich from any constraints on how they are able to shape the world can definitely be a big problem.
There's an implied assumption that when you lose parts of society through a bottleneck that you can always recreate them with high fidelity. It seems plausible that some bottleneck events could "limit humanity's potential", since choices may rely on those lost values, and not all choices are exchangeable in time. (This has connections both to the long reflection and to the rich shaping the world in their own image).
As an aside, the bottleneck paper you're referring to is pretty contentious. I personally find it unlikely that no other demographic model detects a bottleneck of >0.99 in the real data, but all of them can do it on simulated data. If such an event did occur in the modern day, the effects would be profound and serious.
There's an implied assumption that when you lose parts of society through a bottleneck that you can always recreate them with high fidelity.
I do think this assumption will become more true in the 21st century, and indeed AI and robotics progress making the assumption more true is the main reason that if a catastrophe happens, this is how it happens.
Re the bottleneck paper, I'm not going to comment any further, since I just wanted to provide an example.
At this point, I wouldn't agree that that is the main concern. I do think it's a solid third place though.
My top two are:
Deliberately harmful use
Some human decides they want to kill some other humans, and an AI enables them to acquire more power to do so. This could scale really fast, and actually give a single human the capability to wipe out nearly all of humanity. I would not like to bet my loved ones lives on there not being some person in the world crazy enough to try this.
Rogue AGI
I think it's possible that some crazy human might decide that AI becoming an independent species is a good thing actually, and deliberately creating and releasing an AI system which is sufficiently capable and general that it manages to take off. There's also the possibility an AI will independently develop Omohundro Drives and escape of its own volition. That seems less likely to me, but the two probabilities sum together, so... pick your nightmare, I guess.
Both of these threats route through the possibility of AI enabling offense-dominant self-replicating weaponry (bioweapons and/or nanotech). Self-replicating weaponry is a particularly dangerous category, because a small actor can cheaply initiate huge irrevocable effects thereby.
My main thought with those two problems is that I agree they are an issue in the world where AGI is avoided for ideological reasons, but it seems like in the world where AGI gets fully developed, they could simply be prevented by having a good AGI monitor everything and nib such dangers in the bud.
Indeed, my main hope for humanity does route through developing a good AGI monitor to prevent these risks. And, conditional on that happening, your described threat would move to top place.
I don't think the route to having a robust worldwide AGI monitor capable of preventing the harms I describe is a safe or smooth one though. That's where I expect most of humanity's risk currently lies.
One could maybe say that our current system is mainly a mixture of capitalism, which leads to the problem I describe, and democratically-governed nation-states with militaries, which leads to the problem you describe. How do we transition our production vs how do we transition our security.
1Hmm. Did you read the comment I linked? I don't place enough predicted risk weight on state actors that they are the reason for my top 2 threats being the top 2. The danger to me is that high-variability of behavior of individual humans, and the extremely low costs to launching a self-replicating weapon, make it so that all of humanity is currently endangered by a single individual bad actor (human or AGI).
Hmm. Did you read the comment I linked?
I took a quick peek at it at first, but now I've read it more properly.
I don't place enough predicted risk weight on state actors that they are the reason for my top 2 threats being the top 2. The danger to me is that high-variability of behavior of individual humans, and the extremely low costs to launching a self-replicating weapon, make it so that all of humanity is currently endangered by a single individual bad actor (human or AGI).
I think the main question is, why would state actors (which currently provide security by suppressing threats) allow this?
I don't believe they currently possess the means to prevent it.
Creating a devastating bioweapon is currently technically challenging, but not resource intensive and not easy for governments to detect. If government policy around biology materials and equipment does not shift dramatically in the coming three years, the technical difficulty will probably continue to drop while no corresponding increase in prevention occurs.
I'm currently engaged in studying AI related Biorisk, so I know a lot of details I cannot disclose about the current threat situation. I will share what I can. https://securebio.org/ai/
I agree that the negative outcomes from technological unemployment do not get enough attention but my model of how the world will implement Transformative AI is quite different to yours.
Our current society doesn't say "humans should thrive", it says "professional humans should thrive"
Let us define workers to be the set of humans whose primary source of wealth comes from selling their labour. This is a very broad group that includes people colloquially called working class (manual labourers, baristas, office workers, teachers etc) but we are also including many people who are well remunerated for their time, such as surgeons, senior developers or lawyers.
Ceteris paribus, there is a trend that those who can perform more valuable, difficult and specialised work can sell their labour at a higher value. Among workers, those who earn more are usually "professionals". I believe this is essentially the same point you were making.
However, this is not a complete description of who society allows to "thrive". It neglects a small group of people with very high wealth. This is the group of people who have moved beyond needing to sell their labour and instead are rewarded for owning capital. It is this group who society says should thrive and one of the strongest predictors of whether you will be a member is the amount of wealth your parents give you.
The issue is that this small group is owns a disproportionate proportion of shares in frontier AI companies.
Assuming we develop techniques to reliably align AGIs to arbitrary goals, there is little reason to expect private entities to intentionally give up power (doing so would be acting contrary to the interests of their shareholders).
Workers unable to compete with artificial agents will find themselves relying on the charity and goodwill of a small group of elites. (And of course, as technology progresses, this group will eventually include all workers.)
Those lucky enough to own substantial equity in AI companies will thrive as the majority of wealth generated by AI workers flows to them.
In itself, this scenario isn't an existential threat. But I suspect many humans would consider their descendants being trapped into serfdom is a very bad outcome.
I worry a focus on preventing the complete extinction of the human race means that we are moving towards AI Safety solutions which lead to rather bleak futures in the majority of timelines.[1]
- ^
My personal utility function considers permanent techno-feudalism forever removing the agency of the majority of humans is only slightly better than everyone dying.
I suspect that some fraction of humans currently alive also consider a permanent loss of freedom to be only marginally better (or even worse) than death.
It's somewhat intentional that I say "professionals" instead of "workers", because if I understand correctly, by now the majority of the workforce in the most developed countries is made up of white-collar workers. I think AI is especially relevant to us because professionals rely almost exclusively on intelligence, knowledge and connections, whereas e.g. cooks also rely on dexterity. (Admittedly, AI will probably progress simultaneously with robots, which will hit people who do more hands-on work too.)
I think in the scenario you describe, one of the things that matters most is how well the police/military can keep up with the AI frontier. If states maintain sovereignty (or at least, the US maintains hegemony), people can "just" implement a wealth tax to distribute the goods from AI.
Under that model, the question then becomes who to distribute the goods to. I guess the answer would end up being "all citizens". Hm, I was about to say "in which case we are back in the "but wouldn't people's capabilities degenerate into vestigiality?" issue", but really if people have a guaranteed source of income to survive, as long as we are only moderately more expensive than the AI and not extremely more expensive, the AI would probably want to offer us jobs because of comparative advantages. Maybe that's the sort of scenario Habryka was getting at...
(Admittedly, AI will probably progress simultaneously with robots, which will hit people who do more hands-on work too.)
This looks increasingly unlikely to me. It seems to me (from an outsider's perspective) that the current bottleneck in robotics is the low dexterity of existing hardware far more than the software to animate robot arms, or even the physics simulation software to test it. And on the flip side current proto-AGI research makes the embodied cognition thesis seems very unlikely.
1I thought job loss was a short-term but not a long-term risk and only a secondary cause of x-risks. Which is why we don't worry about it around here much. FWIW being enslaved permanently doesn't sound better to me than being extinct, so I wouldn't frame that distinction as a problem.
I thought the standard plan was: we figure out how to ensure that an aligned AGI/ASI takes over; it helps humans because it wants to. It's way better than us at making stuff, so keeping humanity in good conditions is actually super easy, barely an inconvenience.
That could be either because we got value alignment right, or we put personal intent aligned AGI into the hands of someone with the common decency to help all of humanity with it. And it/they wisely decided to prevent the creation of other AGI.
The ASI makes sub-sentient safe minds to do the grunt work; humanity's biggest challenge is deciding what to do with unlimited time and vast resources.
Anyone who's understood a Burning Man event knows that humans have a blast working on collaborative projects, even when they don't need to be done at all, so I have no concerns that we'll all get sad without necessary or "meaningful" work to do.
So job loss seems like a problem only if it makes us freak out and do something stupider.
Perhaps you're thinking of the scenario where AGI takes over the economy but doesn't otherwise directly care about humans. I think job loss is a weird way to frame humans being utterly obsoleted and having their oxygen eventually "purchased" for better uses. AGI that doesn't care about humans but does care about property rights would be strange. FWIW, I think people sometimes see AGI proliferation as a potential good outcome, but that's exactly wrong; the multipolar scenario with personal intent aligned AGI serving different competing masters seems likely to result in doom, too.
I thought job loss was a short-term but not a long-term risk and only a secondary cause of x-risks. Which is why we don't worry about it around here much. FWIW being enslaved permanently doesn't sound better to me than being extinct, so I wouldn't frame that distinction as a problem.
The AI wanting to permanently enslave us would change the game board in poorly explored ways. I could imagine that e.g. it would be more plausible that it would reliably form concepts of human values, which could be used to align it to prevent slavery. Also its control ability as a slavemaster would likely vastly exceed that of human slavemasters, meaning it probably wouldn't need stuff like violence or keeping us fearful in order to control us. I don't have a clear idea of what it would look like, partly because the scenario is not realistic since the AI would realistically have an absolute economic advantage over us and therefore would rather want us dead than slaves.
I thought the standard plan was: we figure out how to ensure that an aligned AGI/ASI takes over; it helps humans because it wants to. It's way better than us at making stuff, so keeping humanity in good conditions is actually super easy, barely an inconvenience.
That is the standard rationalist plan, but "take over the world, for the greater good (of course)" is such a classic villain move that we shouldn't be surprised when all the ways we've explored seem to lead to horrible outcomes.
Anyone who's understood a Burning Man event knows that humans have a blast working on collaborative projects, even when they don't need to be done at all, so I have no concerns that we'll all get sad without necessary or "meaningful" work to do.
Idk, Burning Man consists of people who have been shaped by society and who go out of their way to participate. I could imagine that without society existing in the background, people would not really be maintaining the ambition or skills necessary to have something like Burning Man exist.
rationalists think that it is a bigger risk that the AI might murder us all, and that automation gives humans more wealth and free time and is therefore good.
I don't think this is really accurate. It's not about "bigger." For humanity, surviving and thriving depend on us overcoming or avoiding all the x-risks. Rather, many rationalists expect everyone-not-dying to be the first ASI safety gate we have to get through, temporally and ontologically (if we're worrying about unemployment, we're alive to do so).
Thanks for writing this, this is something I have thought about before trying to convince people who are more worried about "short-term" issues to take the "long-term" risks seriously. Essentially, one can think of two major "short-term" AI risk scenarios (or, at least "medium-term" ones that "short-term"ists might take seriously), essentially corresponding to the prospects of automating the two factors of production:
- Mass technological unemployment causing large swathes of workers to become superfluous and then starved out by the now AI-enabled corporations (what you worry about in this post)
- AI increasingly replacing "fallible" human decision-makers in corporations if not in government, pushed by the necessity to maximize profits to be unfettered by any moral and legal norm (even more so than human executives are already incentivized to be; what Scott worries about here)
But if 1 and 2 happens at the same time, you've got your more traditional scenario: AI taking over the world and killing all humans as they have become superfluous. This doesn't provide a full-blown case for the more Orthodox AI-go-FOOM scenario (you would need ), but at least serve as a case that one should believe Reform AI Alignment is a pressing issue, and those who are convinced about that will ultimately be more likely to take the AI-go-FOOM scenario seriously, or at least operationalize one's differences with believers in only object-level disagreements about intelligence explosion macroeconomics, how powerful is intelligence as a "cognitive superpower", etc. as opposed to the tribalized meta-level disagreements that define the current "AI ethics" v. "AI alignment" discourse.
It seems to me that when normal people are concerned about AI destroying their life, they are mostly worried about technological unemployment, whereas rationalists think that it is a bigger risk that the AI might murder us all, and that automation gives humans more wealth and free time and is therefore good.
I'm not entirely unsympathetic to the rationalist position here. If we had a plan for how to use AI to create a utopia where humanity could thrive, I'd be all for it. We have problems (like death) that we are quite far from solving, and which it seems like a superintelligence could in principle quickly solve.
But this requires value alignment: we need to be quite careful what we mean by concepts like "humanity", "thrive", etc., so the AI can explicitly maintain good conditions. What kinds of humans do we want, and what kinds of thriving should they have? This needs to be explicitly planned by any agent which solves this task.
Our current society doesn't say "humans should thrive", it says "professional humans should thrive"; certain alternative types of humans like thieves are explicitly suppressed, and other types of humans like beggars are not exactly encouraged. This is of course not an accident: professionals produce value, which is what allows society to exist in the first place. But with technological unemployment, we decouple professional humans from value production, undermining the current society's priority of human welfare.
This loss is what causes existential risk. If humanity was indefinitely competitive in most tasks, the AIs would want to trade with us or enslave us instead of murdering us or letting us starve to death. Even if we manage to figure out how to value-align AIs, this loss leads to major questions about what to value-align the AIs to, since e.g. if we value human capabilities, the fact that those capabilities become uncompetitive likely means that they will diminish to the point of being vestigial.
It's unclear how to solve this problem. Eliezer's original suggestion was to keep humans more capable than AIs by increasing the capabilities of humans. Yet even increasing the capabilities of humanity is difficult, let alone keeping up with technological development. Robin Hanson suggests that humanity should just sit back and live off our wealth as we got replaced. I guess that's the path we're currently on, but it is really dubious to me whether we'll be able to keep that wealth, and whether the society that replaces us will have any moral worth. Either way, these questions are nearly impossible to separate from the question of, what kinds of production will be performed in the future?