Epistemic Status

For a "first sketch", this is probably fine.

[This may be superseded by a later post].

Acknowledgements

I'd like to thank "JustinMills", "davidad" and others for their assistance reviewing drafts of this post.

Objectives

My main objectives with this piece are to answer the following questions:

What do I mean by "AI takeoff dynamics"?
Why is the concept useful?
What are its core inquiries?
How do we investigate it?

Introduction

This will be a purely personal approach.

I will present the problem as I see it. Explain why I think it's a useful thing to study, what I believe to be the core inquiries of the field and how best I believe we can investigate it.

I will often say just "takeoff dynamics"; the "artificial intelligence" should be understood as implicit.

Many sections of this piece provide a high-level summary of the topic and link to another post that explores the topic at length. This was needed because the post would have grown excessively long otherwise.

An Informal Presentation of Takeoff Dynamics

To a first approximation, you could view the field as an attempt to identify what kind of takeoff scenario for artificial intelligence we should anticipate, and to what degree. One might be asking the following questions:

"Will we have a slow takeoff?"
- "How slow will it be?
"Or will it be a fast takeoff instead?"
- "How fast will it be?"
"Or perhaps it's a 'medium' takeoff?"
"What if it's slower in some places and faster in others?"

These are the vague questions I had in mind when I first started thinking about "takeoff dynamics".

(The last couple of questions are a bit farcical, to explain why I'm dissatisfied with framing the inquiry in terms of the first two.)

More succinctly, I suppose you could view "takeoff dynamics" as an investigation into what the trajectory of AI capabilities development might (will?) look like. An attempt to forecast future AI development by extrapolating (a) curve(s) forward in time. Taking this view, takeoff dynamics can be considered a particular technique for forecasting AI progress.

This is what I think the common parlance meaning of "AI takeoff dynamics" is (at least in the LessWrong [and adjacent] sphere[s]). It was starting from this vague notion of "trajectory of AI capability development" that I began thinking on the matter.

AI takeoff in the common parlance is commonly understood to refer to the future trajectory of AI development after it reaches a certain level of cognitive capabilities (generally parity with humans). I do not necessarily impose the constraint of "post human parity" in my inquiries into "takeoff dynamics".

However, it's fine to assume that by "takeoff" I'm referring to the exact same phenomenon described on the page for the associated tag.

I will eventually try to present a more concrete formulation of "takeoff dynamics", but first, I'd like to explore why we would care.

Interlude

To avoid debates and/or disagreements over concrete timelines, I'll refrain from specifying any concrete values for units of time in the remainder of this post. I'll use variable names and an arbitrary unit of time whenever such need arises.

Feel free to substitute your preferred values.

Some Core Premises

[Entire Section To Be Rewritten]

In order to explain why investigating takeoff dynamics is valuable, I need to lay out some underlying premises. If one accepts these premises as true, then the value of takeoff dynamics research becomes readily apparent.

The premises:

Artificial Superintelligence (ASI) is permitted by the "fundamental laws" of reality.
Misaligned ASI is an existential threat to humanity.
Aligning ASI is "difficult".

I will briefly try to explain what I mean by those premises but will not devote undue effort towards defending them (a proper defence is worth an extensive essay in its own right).

Feel free to skip the rest of the section if you already find yourself in agreement with these premises.

ASI is Permitted by the Fundamental Laws

By the fundamental laws, I refer to the laws or theorems which may impose harsh, impossible to overcome – fundamental — limits on what cognitive capabilities can be realised in our universe.

A non-exhaustive list of some fields of study whose principles, laws & theorems may bear here include:

Physics
- Thermodynamics of reversible and irreversible computing
Computer Science
- Computational complexity
- Computational intractability
Mathematical optimisation
- E.g., "No Free Lunch" theorems and similar
Statistical and Algorithmic Information theory
Statistical and Algorithmic Learning theory
(Algorithmic) Decision & Game theory

And by "Artificial Superintelligence", I refer to an AI system in possession of cognitive capabilities so advanced that they confer (or enable the system to acquire within years [ $k$ is some length of time suitably short that we cannot adequately respond]) "decisive strategic advantage" over the extant human civilisation (at the time the AI emerges).

The above definition is very clunky, so if we're being liberal with our definitions, you might instead define ASI as:

An AI system with decisive strategic advantage.

I'd like to stress that when I speak of "superintelligence", I am not primarily concerned with the AI's cognitive capabilities, but its real-world capabilities. I am most concerned over whether the system has decisive strategic advantage or not. Its cognitive capabilities are presumably what grant it said real world capabilities, but a par human AI with decisive strategic advantage is no less dangerous for it.

On occasions when I simply want to refer to an AI system in possession of "high cognitive powers", I would refer to it as a "strongly superhuman intelligence" (SSI).

The premise can be stated less ambiguously as:

The fundamental laws do not forbid the emergence of an artificial intelligence that attains decisive strategic advantage over the extant human civilisation at the time it emerges.

(This clunky formulation is unavoidable as what constitutes "decisive strategic advantage" is a moving target and depends heavily on the resilience and infrastructure of the civilisation the AI emerged in).

Misaligned ASI is an Existential Threat to Humanity

By "misaligned", I simply mean any AI that isn't robustly aligned. Operationally, I mean any AI that does not:

Explicitly prioritise human wellbeing and welfare (sensibly construed).
Robustly target human values (sensibly construed).

The above operationalisation assumes sensible and commonsense notions of "human wellbeing", "human welfare" and "human values". These are hard concepts to define in a way that I would fully endorse on deep reflection, so I'll just appeal to our shared vague concepts of them. See also:

The Fragile Purposes sequence
- Especially "The Hidden Complexity of Wishes"
The Value Theory sequence
- Especially "Value is Fragile"

An ASI that isn't robustly aligned poses a credible existential threat to us. Not because it hates us, desires our suffering, or otherwise bear us any malign intent, but because it does not care for our wellbeing or flourishing. We may be an obstacle to whatever values it does care to realise and if it has power over us, it may remove us, the same way we removed several other obstacles in our path:

Aligning ASI is "Difficult"

We do not currently know of any way to align AIs that are robust to arbitrary capability amplification (that is, techniques that will scale up even to the strongly superhuman domain). I am under the impression that we don't know how to robustly align even considerably less powerful systems.

This difficulty has persisted even despite several highly intelligent people working on the problem for over a decade. It's not clear when we'll solve the problem. Some notable domain experts are quite pessimistic.

Motivations

Given the existential threat posed by ASI and the difficulty of aligning ASI, we might want to ask: "well, how much time do we have left?"

This is one reason the trajectory of AI capabilities development is an important thing to investigate. Why it really matters.

Specifically, I want to be able to give (tight) lower bounds on the time periods between different milestones of AI development. That is, I would like to be able to make statements such as:

Given that it took $x$ time to transition from 2022 SOTA to near-human systems, it will take at least $y$ time to transition from near-human systems to par human.
Given that it took $v$ time to transition from the near-human domain to the par human domain, it will take at least $w$ time to transition from the par human domain to a superhuman domain.

I think being able to make such statements is useful in a few ways:

Straightforward safety guarantees
- We have at least this much time left until "catastrophe".
Informing policy choices to mitigate risks from development of advanced AI systems
Informing alignment approaches and/or safety strategies

The first use case being self-evident, I will try to elaborate in brief how takeoff dynamics research could satisfy the latter two use cases.

Informing AI Safety

[I may edit or rewrite this subsection later]

The ideal policies to pursue under fast, medium, and slow takeoff scenarios are markedly different.

The concrete timelines to various capability milestones impact what safety research directions to prioritise.

If par human AI is imminent (e.g. $< n$ years) and takeoff won't be "slow" (e.g. $< m$ years between par human AI and SSI), then alignment strategies that are not immediately applicable to deep learning like models should be heavily deprioritised (a reason not to totally abandon them is the possibility of coordinated slowdowns and to hedge against the fact that we may be mistaken on timelines).

It does not matter if you have a comprehensive theory of how to build aligned artificial intelligence $x$ years after the first SSI arrives. It's too late to apply your comprehensive theory to the SSI, so it might as well be worthless. We are either already saved or already doomed by then.

Similarly, it doesn't matter if you have a comprehensive theory of how to build aligned AI $2$ years before the first SSI arrives if said theory is only applicable to purely symbolic/logical/casual/etc. models (GOFAI), the first SSI is an artificial neural network (ANN), and porting your theory to ANNs takes $3$ years.

What alignment strategies and research directions it even makes sense to pursue depend strongly on:

How much time do we need to make said strategies useful?
How much time do we have left until catastrophe if we can't figure out alignment?

Under certain beliefs around takeoff scenarios, some alignment approaches are in all likelihood "too late" (or otherwise inapplicable) and should be deprioritised.

If you expect the first AGIs (and SSIs) to look more like deep learning systems than like GOFAI, then rigorous mathematical theories that are only applicable to GOFAI like systems should be heavily deprioritised.

A good rule of thumb is that the shorter timelines are to human level AI [HLAI], the greater the resemblance will be between the first HLAI and the current state of the art systems. On the other hand, the longer timelines are to HLAI, the greater the differences will be between the first HLAI and current SOTA.

Longer timelines have more opportunity for paradigm shifts.

An underlying intuition to translate the rule of thumb to AI safety approaches:

More paradigmatic alignment strategies have limited applicability to other paradigms but cover the paradigm they target to greater depth.
More general alignment strategies have broader coverage of diverse paradigms but have limited depth when applied to any particular paradigm.

("Paradigm" here being paradigms of AI development)

The above intuition could be simplified to:

If we knew what the first HLAI/SSI systems would look like, we could do a better job at designing alignment strategies.

I think if we were confident in:

Shorter timelines to HLAI/fast takeoff we should focus more on SOTA like paradigmatic approaches.
- Prosaic ML like strategies is more valuable on shorter timelines
Longer timelines to HLAI/slow takeoff we should focus more on less paradigmatic /more general approaches.
- Fundamental theory/first principles work is more valuable on longer timelines

Informing AI Governance

[I may edit or rewrite this subsection later]

Likewise, what governance approaches are tenable also depends on how much time we have left.

A non-exhaustive list of some relevant questions to illustrate how takeoff speeds bear on governance:

Do we have enough time to draw up international treaties & arms controls?
Do we have enough time to monitor and regulate GPUs/ASICs/other computational resources the way we do e.g., uranium enrichment?
Do we have enough time to set up an AI safety equivalent of the Cybersecurity and Infrastructure Security Agency?
Do we have enough time to establish industry journals, protocols, conferences, & other infrastructure of the cybersecurity industry?
Do we have enough time for meaningful (and helpful) legislation?

(I have a habit of viewing governance around alignment/safety via analogy to cybersecurity. I apologise if this is completely off base or otherwise unhelpful).

Under more gradual takeoff scenarios (e.g., we spend several years in the near human domain before HLAI) accompanied with broad deployment of advanced AI technologies (this seems to be the status quo under which we operate), we can expect to see several significant alignment failures in "the wild". Said failures could serve as a "fire alarm" to trigger collective action/coordination around norms/protocols/rules for responsible deployment of AI technologies and bolster support for other governance proposals.

If the governance folks had a better idea of what the timelines to different milestones looked like (or confident lower/upper bounds), they could better prioritise the approaches that have the highest likelihood of success in the "mainline" (most likely pathway of AI development).

I've been informed by people who work in AI governance/strategy that accurate models of timelines could be helpful for:

Informing coordinated "slowdown" policies
- Determining credible "un-slowdown" conditions
Enabling sophisticated development/deployment strategies based on control theory-esque feedback loops
- Enabling incremental development at each stage for high assurance/very robust systems before advancing towards the next stage
Knowing that it's nontrivial to go from peak human systems to strongly superhuman systems would make such plans palatable/credible

Some Necessary(?) Terminology

There are some terms/concepts that I'll use later in this piece to formulate my presentation of takeoff dynamics. I'd like to disambiguate them here.

Computational Resources

The various resources that can be utilised to increase the cognitive capabilities of an AI system.

A non-exhaustive list:

Training compute
Inference compute
Training data
Inference data
Accessible memory
Bandwidth
Energy/power during training
Energy/power during inference
Etc.

(I distinguish training from inference because the resource utilisation during those two phases may vary considerably. Though for some systems, the training/inference distinction may not be sharp).

I cover this in more detail in: "Preliminary Thoughts on Computational Resources".

Cognitive Capabilities

The various aspects of cognition. Diverse ways in which intelligence manifests itself. Concrete skills/tools that are part of what we call "intelligence" (broadly construed).

A non-exhaustive list:

Prediction
Planning
Compression
Deduction
Induction
Abstraction
Other symbolic reasoning
Imagination
Concept synthesis
Concept generation
Broad pattern matching
Learning
Meta learning
Etc.
Aggregations of other cognitive capabilities
"General intelligence"

I cover this in more detail in: "Preliminary Thoughts on Cognitive Capabilities".

Cognitive Intervals

The general range of cognitive capabilities an agent belongs in.

For convenience's sake (and because they are the only example of a "general intelligence" that we are aware of), I'll use humans as a yardstick.

Some "cognitive intervals" that seem particularly pertinent:

Subhuman to near human
Near human to beginner human
Beginner human to median human professional
Median human professional to expert human
Expert human to superhuman
Superhuman to strongly superhuman
Contractions of previous intervals
Aggregations of prior intervals

"Human" in the intervals mentioned above means something like "median human adult".

I cover this in more detail in: "Preliminary Thoughts on Cognitive Intervals".

Economic Resources

[To Be Written]

I cover this in more detail in: "Preliminary Thoughts on Economic Resources".

"Real World" Capabilities

I distinguish between an agent's intelligence — its cognitive capabilities [broadly construed] — and its ability; its potential to effect changes in its external environment. Cognitive capabilities describe an agent's internal state, its reasoning, world model, knowledge acquisition and generation, etc. "Ability" describes an agent's actions. What changes can the agent reliably cause in its environment, via the actions available to it?

I stress this difference, because in my experience some people seem to conflate the two, and I have found that confusion unhelpful and a hindrance to clear thinking about artificial intelligence.

By "real world capabilities", I refer to an agent's ability to effect changes in the "real world" environment. To effect change in base reality with all its unbounded choice.

There are many kinds of capabilities an agent might have in the real world.

Rather than trying to present a comprehensive taxonomy of all real-world capabilities, I will classify them based on what the primary "interface" for utilising that capability is.

Humans (and groups thereof)
Human infrastructure
Physics/Bare reality

I explore this concept in more detail in "Preliminary Thoughts on 'Real World Capabilities'".

Marginal Returns

Per Wikipedia:

Marginal Return is the rate of return for a marginal increase in investment; roughly, this is the additional output resulting from a one-unit increase in the use of a variable input, while other inputs are constant.

I explore this concept (and how it bears on Takeoff Dynamics) in more detail in "Preliminary Thoughts on 'Marginal Returns'".

Why Marginal Returns?

Why should we use marginal returns as our primary instrument for investigating takeoff dynamics?

[To Be Written]

Economics of AI Development

Detailed economic models of AI development across different cognitive intervals. Some economic considerations relevant to AI development that could bear on Takeoff Dynamics.

[To Be Written]

I cover this in more detail in: "Preliminary Thoughts on the Economics of AI Development".

The Core Inquiries

We now have all the necessary apparatus to present the core inquiries of Takeoff Dynamics.

[To Be Written]

Some Important Marginal Returns

What marginal returns are most important to investigating takeoff dynamics?

Hierarchical Marginal Returns

[To Be Written]

Specific Questions

Marginal returns to cognitive capabilities from the investment of computational resources
- How do marginal returns to cognitive capabilities behave as a function of different computational resources?
- How much smarter can you make an AI system by throwing more compute at it during training or inference?
Marginal returns to cognitive capabilities from the investment of cognitive resources
1. Marginal returns on algorithmic/architectural innovation
  - How much smarter can you make a system by improving its architecture/algorithms?
  - Relevant to:
    - Humans designing new AI architectures/algorithms
    - AI contributing to AI research and development
    - AIs designing successor systems
      - Recursive self-improvement
2. Marginal returns on population
  - How much smarter can you make a system by adding more of it?
  - Relevant to:
    - Swarm intelligence
    - Hive minds
    - Committees
    - Corporations
    - Other strategies for collective intelligences
3. Marginal returns on size
  - How much smarter can you make a system by making it bigger?
  - Larger cognitive engines ("brains")
    - Brains with more synaptic connections/neurons
    - ML models with more parameters/hyperparameters
Marginal returns to computational resources from economic investment
- How much more computational resources can you buy by throwing more money directly at computational resources?
  - What’s the behaviour of the cost curves of computational resources?
    - Economies of scale
    - Production efficiencies
- How much more computational resources can you buy by throwing more money at cognitive labour to acquire computational resources?
  - How does investment of cognitive labour affect the cost curves of computational resources?
  - E.g., innovation to raise efficiency of production
Marginal returns to real world capabilities from the investment of cognitive capabilities
- How much more capability in the real world does being smarter buy you?
Marginal returns to real world capabilities from investment of economic resources via cognitive investment
- Determines whether investment in cognitive amplification is an attractive proposition economically
  - If investment in other capabilities (e.g., acquiring more energy) had better returns on investment, economic resources available for cognitive amplification would be lower.

[To be expanded upon]

Evaluating Results

How do we evaluate results of takeoff dynamics research?

[To Be Written]

Retrodictions

[To Be Written]

Examples

[To Be Written]

Required Tools

Tools, instruments, and machinery we need for investigating takeoff dynamics.

[To Be Written]

Next Steps

[To Be Written]

Theory

[To Be Written]

I cover this in more detail in: "Preliminary Thoughts on Theoretical Foundations of Takeoff Dynamics".

Experiments

[To Be Written]

I cover this in more detail in: "Preliminary Thoughts on Experimental Design for Takeoff Dynamics".