Ajeya Cotra

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

I think that in the coming 15-30 years, the world could plausibly develop “transformative AI”: AI powerful enough to bring us into a new, qualitatively different future, via an explosion in science and technology R&D. This sort of AI could be sufficient to make this the most important century of all time for humanity. The most straightforward vision for developing transformative AI that I can imagine working with very little innovation in techniques is what I’ll call human feedback[1] on diverse tasks (HFDT): > Train a powerful neural network model to simultaneously master a wide variety of challenging tasks (e.g. software development, novel-writing, game play, forecasting, etc) by using reinforcement learning on human feedback and other metrics of performance. HFDT is not the only approach to developing transformative AI,[2] and it may not work at all.[3] But I take it very seriously, and I’m aware of increasingly many executives and ML researchers at AI companies who believe something within this space could work soon. Unfortunately, I think that if AI companies race forward training increasingly powerful models using HFDT, this is likely to eventually lead to a full-blown AI takeover (i.e. a possibly violent uprising or coup by AI systems). I don’t think this is a certainty, but it looks like the best-guess default absent specific efforts to prevent it. More specifically, I will argue in this post that humanity is more likely than not to be taken over by misaligned AI if the following three simplifying assumptions all hold: 1. The “racing forward” assumption: AI companies will aggressively attempt to train the most powerful and world-changing models that they can, without “pausing” progress before the point when these models could defeat all of humanity combined if they were so inclined. 2. The “HFDT scales far” assumption: If HFDT is used to train larger and larger models on more and harder tasks, this will eventually result in models that can autonomou

373Jul 18, 2022

Ajeya Cotra

Message

3172

723

239

Survey on the acceleration risks of our new RFPs to study LLM capabilities

My team at Open Philanthropy just launched two requests for proposals: * Proposals to create benchmarks measuring how well LLM agents (like AutoGPT) perform on difficult real-world tasks, similar to recent work by ARC Evals.[1] * Proposals to study and/or forecast the near-term real-world capabilities and impacts of LLMs and...

Nov 10, 202329

AI Timelines

Introduction How many years will pass before transformative AI is built? Three people who have thought about this question a lot are Ajeya Cotra from Open Philanthropy, Daniel Kokotajlo from OpenAI and Ege Erdil from Epoch. Despite each spending at least hundreds of hours investigating this question, they still still...

Nov 10, 2023302

New roles on my team: come build Open Phil's technical AI safety program with me!

Open Phil announced two weeks ago that we’re hiring for over 20 roles across our teams working on global catastrophic risk reduction — and we’ll answer questions at our AMA starting tomorrow. Ahead of that, I wanted to share some information about the roles I’m hiring for on my team...

Oct 19, 202383

New blog: Planned Obsolescence

Kelsey Piper and I just launched a new blog about AI futurism and AI alignment called Planned Obsolescence. If you’re interested, you can check it out here. Both of us have thought a fair bit about what we see as the biggest challenges in technical work and in policy to...

Mar 27, 202396

Two-year update on my personal AI timelines

I worked on my draft report on biological anchors for forecasting AI timelines mainly between ~May 2019 (three months after the release of GPT-2) and ~Jul 2020 (a month after the release of GPT-3), and posted it on LessWrong in Sep 2020 after an internal review process. At the time,...

Aug 2, 2022293

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

Jul 18, 2022373

ARC's first technical report: Eliciting Latent Knowledge

ARC has published a report on Eliciting Latent Knowledge, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and...

Dec 14, 2021230

Load More (7/16)

LESSWRONG
LW

LESSWRONG
LW

Ajeya Cotra

Ajeya Cotra

Ajeya Cotra

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

AI Timelines

Two-year update on my personal AI timelines

ARC's first technical report: Eliciting Latent Knowledge

Ajeya Cotra

Survey on the acceleration risks of our new RFPs to study LLM capabilities

AI Timelines

New roles on my team: come build Open Phil's technical AI safety program with me!

New blog: Planned Obsolescence

Two-year update on my personal AI timelines

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

ARC's first technical report: Eliciting Latent Knowledge

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

AI Timelines

Two-year update on my personal AI timelines

ARC's first technical report: Eliciting Latent Knowledge

Survey on the acceleration risks of our new RFPs to study LLM capabilities

AI Timelines

New roles on my team: come build Open Phil's technical AI safety program with me!

New blog: Planned Obsolescence

Two-year update on my personal AI timelines

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

ARC's first technical report: Eliciting Latent Knowledge