David Scott Krueger (formerly: capybaralet)

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Full version on arXiv | X Executive summary AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment. This loss of human influence will be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship. A gradual loss of control of our own civilization might sound implausible. Hasn't technological disruption usually improved aggregate human welfare? We argue that the alignment of societal systems with human interests has been stable only because of the necessity of human participation for thriving economies, states, and cultures. Once this human participation gets displaced by more competitive machine alternatives, our institutions' incentives for growth will be untethered from a need to ensure human flourishing. Decision-makers at all levels will soon face pressures to reduce human involvement across labor markets, governance structures, cultural production, and even social interactions. Those who resist these pressures will eventually be displaced by those who do not. Still, wouldn't humans notice what's happening and coordinate to stop it? Not necessarily. What makes this transition particularly hard to resist is that pressures on each societal system bleed into the others. For example, we might attempt to use state power and cultural attitudes to preserve human economic power. However, the economic incentives for companies to replace humans with AI will also push them to influence states and culture to support this change, using their growing economic power to shape both policy and public opinion, which will in t

183Jan 30, 2025

An Update on Academia vs. Industry (one year into my faculty job)

122Sep 3, 2022

"Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities)

112Mar 18, 2023

Causal confusion as an argument against the scaling hypothesis

86Jun 20, 2022

David Scott Krueger (formerly: capybaralet)

Message

https://twitter.com/DavidSKrueger
https://www.davidscottkrueger.com/
https://therealartificialintelligence.substack.com/p/the-real-ai-deploys-itself

2644

599

475

11y

[USA Today op-ed]: No, AI isn't inevitable. We should stop it while we can.

Feb 2412

Two Aspects of Situational Awareness: World Modelling & Indexical Information

I'm writing this post to share some of my thinking about situational awareness, since I'm not sure others are thinking about it this way. For context, I think situational awareness is a critical part of the case for rogue AI and scheming-type risks. But incredibly, it seems to have been...

Jan 740

High-level approaches to rigor in interpretability

There are three broad types of approach I see for making interpretability rigorous. I've put them in ascending order of how much assurance I think they can provide. I think they all have pros and cons, and am generally in favor of rigor. 1. (weakest) Practical utility: Does this interpretability...

Dec 8, 202524

A few quick thoughts on measuring disempowerment

People want to measure and track gradual disempowerment. One issue with a lot of the proposals I've seen is that they don't distinguish between empowering and disempowering uses of AI. If everyone is using AI to write all of their code, that doesn't necessarily mean they are disempowered (in an...

Dec 8, 202530

AI is not inevitable.

AI companies are explicitly trying to build AIs that are smarter than humans, despite clear signs that it might lead to human extinction. It will be tragic and ironic if humanity’s largest project ever is an all-out race to destroy ourselves. But can we really stop building more and more...

Nov 7, 202529

My new nonprofit Evitable is hiring.

https://evitable.com/ Our mission is to inform and organize the public to confront societal-scale risks of AI, and put an end to the reckless race to develop superintelligence. We're hiring for 3 roles: 1) Operations Associate or Head of Operations 2) Communications Associate or Head of Communications 3) Chief of Staff...

Nov 7, 202574

Antisocial media: AI’s killer app?

Plans for what to do with artificial general intelligence (“AGI”) have always been ominously vague… “Solve intelligence” and “use [it] to solve everything else” (Google DeepMind). “We’ll ask the AI” (OpenAI). One money-making idea is starting to crystallize: Replacing your friends with fake AI people who manipulate you and sell...

Oct 3, 202535

Load More (7/70)

LESSWRONG
LW

LESSWRONG
LW

David Scott Krueger (formerly: capybaralet)

David Scott Krueger (formerly: capybaralet)

David Scott Krueger (formerly: capybaralet)

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

An Update on Academia vs. Industry (one year into my faculty job)

"Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities)

Causal confusion as an argument against the scaling hypothesis

David Scott Krueger (formerly: capybaralet)

[USA Today op-ed]: No, AI isn't inevitable. We should stop it while we can.

Two Aspects of Situational Awareness: World Modelling & Indexical Information

High-level approaches to rigor in interpretability

A few quick thoughts on measuring disempowerment

AI is not inevitable.

My new nonprofit Evitable is hiring.

Antisocial media: AI’s killer app?

[USA Today op-ed]: No, AI isn't inevitable. We should stop it while we can.

Two Aspects of Situational Awareness: World Modelling & Indexical Information

High-level approaches to rigor in interpretability

A few quick thoughts on measuring disempowerment

AI is not inevitable.

My new nonprofit Evitable is hiring.

Antisocial media: AI’s killer app?

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

An Update on Academia vs. Industry (one year into my faculty job)

"Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities)

Causal confusion as an argument against the scaling hypothesis