Three Types of Intelligence Explosion

Tom Davidson; wdmacaskill

Some quick (and relatively minor) notes:

I expect that full stack intelligence explosion could look more like "make the whole economy bigger using a bunch of AI labor" rather than specifically automating the chip production process. (That said, in practice I expect explicit focused automation of chip production to be an important part of the picture, probably the majority of the acceleration effect.) Minimally, you need to scale up energy at some point.
- Focusing on the whole economy is closer to the perspective (I think) of some people from Epoch like Tamay, Matthew, and Ege.
You talk about "chip technology" feedback loop as taking months, but presumably improvements to ASML take longer as they often require building new fabs?
The 6 OOM limit for chip technology is based on limits to FLOP/joule, but currently, we're not limited by energy prices/supply as much we we're limited by chip cost. So, in principle you could improve chip technology by reducing the cost of manufactoring. I think this maybe gets you an extra few OOMs, though it's somewhat unclear how to do the accouting between this and scaling up chip production. When analyzing the cumulative limits this sort of question doesn't matter (as the overall limit can be assessed by just doing intelligence/flop * flop/joule * maximum joules), but when breaking down how much progress is possible from each component, then the flop/joule abstraction doesn't really cleanly map onto an area like "chip technology".

I expect that full stack intelligence explosion could look more like "make the whole economy bigger using a bunch of AI labor" rather than specifically automating the chip production process. (That said, in practice I expect explicit focused automation of chip production to be an important part of the picture, probably the majority of the acceleration effect.) Minimally, you need to scale up energy at some point.

Agreed on the substance, we just didn't explain this well.

You talk about "chip technology" feedback loop as taking months, but presumably improvements to ASML take longer as they often require building new fabs?

Agreed!

Re Flop/joule also agree on the substance -- we went with FLOP/joule bc we wanted a clean estimate for the OOMs before reaching limits for each factor. I believe our estimate of the total OOMs to limits (including both chip tech and chip production) is right, but you're right that there are ways to intutively improve chip tech that don't increase FLOP/joule

[-]mishka6mo90

Thanks for writing this.

We estimate that before hitting limits, the software feedback loop could increase effective compute by ~13 orders of magnitude (“OOMs”)

This is one place where I am not quite sure we have the right language. On one hand, the overall methodology pushes us towards talking in terms of "orders of magnitude of improvement", a factor of improvement which might be very large, but it is a large constant.

On the other hand, algorithmic improvements are often improvements in algorithmic complexity (e.g. something is no longer exponential, or something has a lower degree polynomial complexity than before, like linear instead of quadratic). Here the factor of improvement is growing with the size of a problem in an unlimited fashion.

And then, if one wants to express this kind of improvement as a constant, one needs to average the efficiency gain over the practical distribution of problems (which itself might be a moving target).^[1]

In particular, one might think about algorithms searching for better architecture of neural machines, or algorithms searching for better optimization algorithms. The complexity improvements in those algorithms might be particularly consequential. ↩︎

[-]PeterMcCluskey6mo60

I agree with most of this, but the 13 OOMs from the the software feedback loop sounds implausible.

From How Far Can AI Progress Before Hitting Effective Physical Limits?:

the brain is severely undertrained, humans spend only a small fraction of their time on focussed academic learning

I expect that humans spend at least 10% of their first decade building a world model, and that evolution has heavily optimized at least the first couple of years of that. A large improvement in school-based learning wouldn't have much effect on my estimate of the total learning needed.

[-]Tom Davidson6mo80

It does sound like a lot -- that's 5 OOMs to reach human learning efficiency and then 8 OOMs more. But when we BOTECed the sources of algorithmic efficiency gain on top of the human brain, it seemed like you could easily get more than 8. But agreed it seems like a lot. Though we are talking about ultimate physical limits here!

Interesting re the early years. So you'd accept that learning from 5/6 could be OOMs more efficient, but would deny that the early years could be improved?

Though you're not really speaking to the 'undertrained' point, which is about the number of params vs data points

[-]Noosphere896mo40

I agree evolution has probably optimized human learning, but I don't think that it's so heavily optimized that we can use it to give a tighter upper bound than 13 OOMs, and the reason for this is I do not believe that humans are in equilibrium, and this means that there are probably optimizations left to discover, so I do think the 13 OOMs number is plausible )with high uncertainty).

Comment below:

https://www.lesswrong.com/posts/DbT4awLGyBRFbWugh/#mmS5LcrNuX2hBbQQE

[-]PeterMcCluskey6mo40

The first year or two of human learning seem optimized enough that they're mostly in evolutionary equilibrium - see Henrich's discussion of the similarities to chimpanzees in The Secret of Our Success.

Human learning around age 10 is presumably far from equilibrium.

I'll guess that I see more of the valuable learning taking place in the first 2 years or so than do other people here.

[-]Noosphere896mo40

I have 2 cruxes here:

I buy Heinrich's theory far less than I used to, because Heinrich made easily checkable false claims that all point in the direction of culture being more necessary for human success.

In particular, I do not buy that humans and chimpanzees are nearly that similar as Heinrich describes, and a big reason for this is that the study that showed that had heavily optimized and selected the best chimpanzees against reasonably average humans, which is not a good way to compare performance if you want the results to generalize.

I don't think they're wildly different, and I'd usually put chimps effective flops as 1-2 OOMs lower, but I wouldn't go nearly as far as Heinrich on the similarities.

I do think culture actually matters, but nowhere near as much as Heinrich wants it to matter.

I basically disagree that most of the valuable learning takes place before age 2, and indeed if I wanted to argue the most valuable point for learning, it would probably be from 0-25 years, or more specifically 2-7 years olds and then 13-25 years old again.

LESSWRONG
LW

LESSWRONG
LW

40

Three Types of Intelligence Explosion

40

Ω 16

40

Ω 16

Abstract

Summary