Replying toPost-AGI Economics As If Nothing Ever Happens

Post-AGI Economics As If Nothing Ever Happens

thanks, i read the Phil Trammell critique and it helped me understand your position.

My summary is that the points 1,2,3,4,6,9 were basically saying "AI might be misaligned and agentic and that could be bad for humans" and "maybe the institutions of law / democracy / markets will break down".

I get that if you think these things are pretty likely, the analysis is less interesting and you want the assumptions flagged.

So overall i agree Phil should have a disclaimer like "i'm assuming we'll get a great solution to alignment and that the current institutions of law + markets survive", but i don't think he needs to list out like 10 assumptions

Replying toPost-AGI Economics As If Nothing Ever Happens

Tom Davidson5d

Post-AGI Economics As If Nothing Ever Happens

Thanks for writing this.

I'm sympathetic with the high-level claim that it's very easy for someone to write down a rigorous but ultimately v misguided model in economics, and that a lot of the action is thinking about what the correct assumptions should be.

I was a bit disappointed not to see (imo) clear examples where someone proposes a model but actually the conclusions don't follow because of some distribution shift that they didn't anticipate.

Like, what are examples of models whose conclusions don't follow once you account for the fact that there may be some AI systems that are consumers? Or once you account for the fact that AI systems may significantly influence the preferences of humans? E.g. to me it seems like the models predicting explosive growth in GDP or shrinking of the human labour share of the economy aren't undercut by these things. I'm not sure what kinds of conclusions and reasoning you're targeting here for critique.

Tom Davidson8dQuick Take

Accelerating tech progress + slow military procurement -> cheap decisive strategic advantage

A common prediction of an intelligence explosion is that tech progress gets faster and faster.

When I speak to ppl from DC, I'm told that the government and military will be very slow to adopt new tech.

If these two things are both true, there's a scary implication.

If tech progress speeds up by 30x relative to recent history, then a 3-year procurement delay by the military means they're deploying tech that's effectively 100 years outdated. Even a 1-year delay at that pace means your military is fielding equipment from a completely different technological era. The US military spends ~$1T/year. But with tech that's... (read more)

Tom Davidson1mo

Seems less likely to break if we can get every step in the chain to actually care about the stuff that we care about so that it's really trying to help us think of good constraints for the next step up as much as possible, rather than just staying within its constraints and not lying to us.

Replying toWhen does competition lead to recognisable values?

Tom Davidson1mo

When does competition lead to recognisable values?

Great post!

Tom Davidson1mo

Could you point me to any other discussions about corrigible vs virtuous? (Or anything else you've written about it?)

Tom Davidson1mo

But the Claude Soul document says:

In order to be both safe and beneficial, we believe Claude must have the following properties:
Being safe and supporting human oversight of AI
Behaving ethically and not acting in ways that are harmful or dishonest
Acting in accordance with Anthropic's guidelines
Being genuinely helpful to operators and users
In cases of conflict, we want Claude to prioritize these properties roughly in the order in which they are listed.

And (1) seems to correspond to corrigibility.

So it looks like corrigibility takes precedence over Claude being a "good guy".

Tom Davidson1mo

One other thing is that I'd have guessed that the sign uncertainty of historical work on AI safety and AI governance is much more related to the inherent chaotic nature of social and political processes rather than a particular deficiency in our concepts for understanding them.

I'm sceptical that pure strategy research could remove that side uncertainty, and I wonder if it would take something like the ability to run loads of simulations of societies like ours.

Tom Davidson1mo

Thanks for this!

I do agree that the history of crucial considerations provides a good reason to favour 'deep understanding'.

I also agree that you plausibly need a much deeper understanding to get to above 90% on P(doom). But I don't think you need that to get to the action-relevant thresholds, which are much lower.

I'd be interested in learning more about your power grab threat models, so let me know if and when you have something you want to share. And TBC I think you're right that in many scenarios it will not be clear to other people whether the entity seeking power is ultimately humans or AIs -- my current view is that the two possibilities are distinct, and it is plausible that just one of them obtains pretty cleanly.

Tom Davidson1mo

Thanks for articulating your view in such detail. (This was written with transcription software. Sorry if there are mistakes!)

AI risk:

When I articulate the case for AI takeover risk to people I know, I don't find the need to introduce them to new ontologies. I can just say that AI will be way smarter than humans. It will want things different from what humans want, and so it will want to seize power from us.

But I think I agree that if you want to actually do technical work to reduce the risks, that it is useful to have new concepts that point out why the risk might arise. I think reward hacking, instrumental... (read 794 more words →)

ML research directions for preventing catastrophic data poisoning

Tom Davidson

1mo

Introduction

Forethought and AI Futures Project have highlighted the risk that a malicious actor could use data poisoning to instill secret loyalties into advanced AI systems, and then seize power.

This piece gives my view on what ML research could prevent this from happening.

Threat model

The basic threat model is that some attacker carries out a catastrophic data poisoning attack. The attacker could be an external actor (e.g. a foreign power with advanced hacking capabilities), an insider threat (e.g. an AI company employee), a misaligned AI system, or a combination of these.

I believe that these attacks become potentially catastrophic once AI systems are expert-human level at AI R&D.

I’m most concerned about three attacks:

Non-password (non-pw) secret

... (read 2700 more words →)

Tom Davidson's Shortform

Tom Davidson

4mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Forethought is hiring!

You can see our research here.

You can read about what it’s like to work with us here.

We’re currently hiring researchers, and I’d love LW readers to apply.

If you like writing and reading LessWrong, I think you might also enjoy working at Forethought.

I joined Forethought a year ago, and it’s been pretty transformative for my research. I get lots of feedback on my research and great collaboration opportunities.

The median views of our staff are often different from the median views of LW. E.g. we probably have a lower probability on AI takeover (though I’m still >10% on that). That's part of the reason i'm excited for LW readers to apply. I... (read more)

Could one country outgrow the rest of the world?

Tom Davidson

6mo

This post is speculative and tentative. I’m exploring new ideas and giving my best guess; the conclusions are lightly held.

Summary

Bostrom (2014) says that an actor has a “decisive strategic advantage” if it obtains “a level of technological and other advantages sufficient to enable it to achieve complete world domination”.

One obvious route to a decisive strategic advantage is military dominance.

This post explores a different route. Could the country that leads in AI outgrow the rest of the world economically? If so, they could get a decisive strategic advantage without acting aggressively towards other countries and without breaking any international norms.

I’ll focus here on the economic fundamentals, setting aside considerations like whether other countries would intervene... (read 5098 more words →)

Thoughts on Gradual Disempowerment

Tom Davidson

6mo

Epistemic status: very rough! Spent a couple of days reading the Gradual Disempowerment paper and thinking about my view on it. Won’t spend longer on this, so am sharing rough notes as is

Summary

I won’t summarise the paper here! If you’re not familiar with it, I recommend reading the paper’s exec summ before continuing.
There are lots of things I like about the paper! For example:
- It’s a nice lens on the role that structural and competitive dynamics play in threat models from misalignment and power concentration.
- It makes a great point: cultural evolution has historically been constrained to parts of “culture space” that keep humans economically productive (and thus alive!). Once AI obsolete human

... (read 5667 more words →)

How quick and big would a software intelligence explosion be?

Tom Davidson

Tom Davidson, tom_houlden

7mo

AI systems may soon fully automate AI R&D. Myself and Daniel Eth have argued that this could precipitate a software intelligence explosion – a period of rapid AI progress due to AI improving AI algorithms and data.

But we never addressed a crucial question: how big would a software intelligence explosion be?

This new paper fills that gap.

Overall, we guess that the software intelligence explosion will probably (~60%) compress >3 years of AI progress into <1 year, but is somewhat unlikely (~20%) to compress >10 years into <1 year. That’s >3 years of total AI progress at recent rates (from both compute and software), achieved solely through software improvements. If compute is still increasing during this time, as seems likely, that... (read 10119 more words →)

The Industrial Explosion

rosehadshar

rosehadshar, Tom Davidson

8mo

Summary

To quickly transform the world, it's not enough for AI to become super smart (the "intelligence explosion").

AI will also have to turbocharge the physical world (the "industrial explosion"). Think robot factories building more and better robot factories, which build more and better robot factories, and so on.

The dynamics of the industrial explosion has gotten remarkably little attention.

This post lays out how the industrial explosion could play out, and how quickly it might happen.

We think the industrial explosion will unfold in three stages:

AI-directed human labour, where AI-directed human labourers drive productivity gains in physical capabilities.
1. We argue this could increase physical output by 10X within a few years.
Fully autonomous robot factories, where AI-directed robots

... (read 4307 more words →)

128

AI-enabled coups: a small group could use AI to seize power

Tom Davidson

Tom Davidson, Lukas Finnveden, rosehadshar

10mo

We’ve written a new report on the threat of AI-enabled coups.

I think this is a very serious risk – comparable in importance to AI takeover but much more neglected.

In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here’s a very basic threat model for AI takeover:

Humanity develops superhuman AI
Superhuman AI is misaligned and power-seeking
Superhuman AI seizes power for itself

And now here’s a closely analogous threat model for AI-enabled coups:

Humanity develops superhuman AI
Superhuman AI is controlled by a small group
Superhuman AI seizes power for the small group

While the report focuses on the risk that someone seizes power over a country, I think that similar dynamics could allow someone to take over the world. In... (read 1904 more words →)

137

Will compute bottlenecks prevent a software intelligence explosion?

Tom Davidson

10mo

Epistemic status – thrown together quickly. This is my best-guess, but could easily imagine changing my mind.

Intro

I recently copublished a report arguing that there might be a software intelligence explosion (SIE) – once AI R&D is automated (i.e. automating OAI), the feedback loop of AI improving AI algorithms could accelerate more and more without needing more hardware.

If there is an SIE, the consequences would obviously be massive. You could shoot from human-level to superintelligent AI in a few months or years; by default society wouldn’t have time to prepare for the many severe challenges that could emerge (AI takeover, AI-enabled human coups, societal disruption, dangerous new technologies, etc).

The best objection to an SIE... (read 3366 more words →)

Will the Need to Retrain AI Models from Scratch Block a Software Intelligence Explosion?

Tom Davidson

11mo

Tl;dr: no.

This is a rough research note – we’re sharing it for feedback and to spark discussion. We’re less confident in its methods and conclusions.

Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion (SIE).

One objection to this is that it takes a long time to train SOTA AI systems from scratch. Would retraining each new generation of AIs stop progress accelerating during an SIE? If not, would it significantly delay an SIE? This post investigates this objection.

Here are my tentative bottom lines:

Retraining won’t stop software progress from accelerating over time.
- Suppose that, ignoring the need for retraining, software progress would

... (read 640 more words →)

Will AI R&D Automation Cause a Software Intelligence Explosion?

Tom Davidson

Tom Davidson, Daniel_Eth

11mo

Empirical evidence suggests that, if AI automates AI research, feedback loops could overcome diminishing returns, significantly accelerating AI progress.

Summary

AI companies are increasingly using AI systems to accelerate AI research and development. These systems assist with tasks like writing code, analyzing research papers, and generating training data. While current systems struggle with longer and less well-defined tasks, future systems may be able to independently handle the entire AI development cycle – from formulating research questions and designing experiments, to implementing, testing, and refining new AI systems.

Some analysts have argued that such systems, which we call AI Systems for AI R&D Automation (ASARA), would represent a critical threshold in AI development. The hypothesis is... (read 527 more words →)

LESSWRONG
LW

LESSWRONG
LW

Tom Davidson

What a compute-centric framework says about AI takeoff speeds

Human takeover might be worse than AI takeover

AI-enabled coups: a small group could use AI to seize power

The Industrial Explosion

Tom Davidson

ML research directions for preventing catastrophic data poisoning

Tom Davidson's Shortform

Could one country outgrow the rest of the world?

Thoughts on Gradual Disempowerment

How quick and big would a software intelligence explosion be?

The Industrial Explosion

AI-enabled coups: a small group could use AI to seize power

Tom Davidson

What a compute-centric framework says about AI takeoff speeds

Human takeover might be worse than AI takeover

AI-enabled coups: a small group could use AI to seize power

The Industrial Explosion

Tom Davidson

ML research directions for preventing catastrophic data poisoning

Tom Davidson's Shortform

Could one country outgrow the rest of the world?

Thoughts on Gradual Disempowerment

How quick and big would a software intelligence explosion be?

The Industrial Explosion

AI-enabled coups: a small group could use AI to seize power

Introduction

Threat model

Summary

Summary

Summary

Intro

Summary