Continuation of:  Recursive Self-Improvement

Constant natural selection pressure, operating on the genes of the hominid line, produced improvement in brains over time that seems to have been, roughly, linear or accelerating; the operation of constant human brains on a pool of knowledge seems to have produced returns that are, very roughly, exponential or superexponential.  (Robin proposes that human progress is well-characterized as a series of exponential modes with diminishing doubling times.)

Recursive self-improvement - an AI rewriting its own cognitive algorithms - identifies the object level of the AI with a force acting on the metacognitive level; it "closes the loop" or "folds the graph in on itself".  E.g. the difference between returns on a constant investment in a bond, and reinvesting the returns into purchasing further bonds, is the difference between the equations y = f(t) = m*t, and dy/dt = f(y) = m*y whose solution is the compound interest exponential, y = e^(m*t).

When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM.  An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely - far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology.  Our present life is no good indicator of things to come.

Or to try and compress it down to a slogan that fits on a T-Shirt - not that I'm saying this is a good idea - "Moore's Law is exponential now; it would be really odd if it stayed exponential with the improving computers doing the research."  I'm not saying you literally get dy/dt = e^y that goes to infinity after finite time - and hardware improvement is in some ways the least interesting factor here - but should we really see the same curve we do now?

RSI is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor.  The advent of human intelligence was a discontinuity with the past even without RSI...

...which is to say that observed evolutionary history - the discontinuity between humans, and chimps who share 95% of our DNA - lightly suggests a critical threshold built into the capabilities that we think of as "general intelligence", a machine that becomes far more powerful once the last gear is added.

This is only a light suggestion because the branching time between humans and chimps is enough time for a good deal of complex adaptation to occur.  We could be looking at the sum of a cascade, not the addition of a final missing gear.  On the other hand, we can look at the gross brain anatomies and see that human brain anatomy and chimp anatomy have not diverged all that much.  On the gripping hand, there's the sudden cultural revolution - the sudden increase in the sophistication of artifacts - that accompanied the appearance of anatomically Cro-Magnons just a few tens of thousands of years ago.

Now of course this might all just be completely inapplicable to the development trajectory of AIs built by human programmers rather than by evolution.  But it at least lightly suggests, and provides a hypothetical illustration of, a discontinuous leap upward in capability that results from a natural feature of the solution space - a point where you go from sorta-okay solutions to totally-amazing solutions as the result of a few final tweaks to the mind design.

I could potentially go on about this notion for a bit - because, in an evolutionary trajectory, it can't literally be a "missing gear", the sort of discontinuity that follows from removing a gear that an otherwise functioning machine was built around.  So if you suppose that a final set of changes was enough to produce a sudden huge leap in effective intelligence, it does demand the question of what those changes were.  Something to do with reflection - the brain modeling or controlling itself - would be one obvious candidate.  Or perhaps a change in motivations (more curious individuals, using the brainpower they have in different directions) in which case you wouldn't expect that discontinuity to appear in the AI's development, but you would expect it to be more effective at earlier stages than humanity's evolutionary history would suggest...  But you could have whole journal issues about that one question, so I'm just going to leave it at that.

Or consider the notion of sudden resource bonanzas.  Suppose there's a semi-sophisticated Artificial General Intelligence running on a cluster of a thousand CPUs.  The AI has not hit a wall - it's still improving itself - but its self-improvement is going so slowly that, the AI calculates, it will take another fifty years for it to engineer / implement / refine just the changes it currently has in mind.  Even if this AI would go FOOM eventually, its current progress is so slow as to constitute being flatlined...

So the AI turns its attention to examining certain blobs of binary code - code composing operating systems, or routers, or DNS services - and then takes over all the poorly defended computers on the Internet.  This may not require what humans would regard as genius, just the ability to examine lots of machine code and do relatively low-grade reasoning on millions of bytes of it.  (I have a saying/hypothesis that a human trying to write code is like someone without a visual cortex trying to paint a picture - we can do it eventually, but we have to go pixel by pixel because we lack a sensory modality for that medium; it's not our native environment.)  The Future may also have more legal ways to obtain large amounts of computing power quickly.

This sort of resource bonanza is intriguing in a number of ways.  By assumption, optimization efficiency is the same, at least for the moment - we're just plugging a few orders of magnitude more resource into the current input/output curve.  With a stupid algorithm, a few orders of magnitude more computing power will buy you only a linear increase in performance - I would not fear Cyc even if ran on a computer the size of the Moon, because there is no there there.

On the other hand, humans have a brain three times as large, and a prefrontal cortex six times as large, as that of a standard primate our size - so with software improvements of the sort that natural selection made over the last five million years, it does not require exponential increases in computing power to support linearly greater intelligence.  Mind you, this sort of biological analogy is always fraught - maybe a human has not much more cognitive horsepower than a chimpanzee, the same underlying tasks being performed, but in a few more domains and with greater reflectivity - the engine outputs the same horsepower, but a few gears were reconfigured to turn each other less wastefully - and so you wouldn't be able to go from human to super-human with just another sixfold increase in processing power... or something like that.

But if the lesson of biology suggests anything, it is that you do not run into logarithmic returns on processing power in the course of reaching human intelligence, even when that processing power increase is strictly parallel rather than serial, provided that you are at least as good as writing software to take advantage of that increased computing power, as natural selection is at producing adaptations - five million years for a sixfold increase in computing power.

Michael Vassar observed in yesterday's comments that humans, by spending linearly more time studying chess, seem to get linear increases in their chess rank (across a wide range of rankings), while putting exponentially more time into a search algorithm is usually required to yield the same range of increase.  Vassar called this "bizarre", but I find it quite natural.  Deep Blue searched the raw game tree of chess; Kasparavo searched the compressed regularities of chess.  It's not surprising that the simple algorithm is logarithmic and the sophisticated algorithm is linear.  One might say similarly of the course of human progress seeming to be closer to exponential, while evolutionary progress is closer to being linear.  Being able to understand the regularity of the search space counts for quite a lot.

If the AI is somewhere in between - not as brute-force as Deep Blue, nor as compressed as a human - then maybe a 10,000-fold increase in computing power will only buy it a 10-fold increase in optimization velocity... but that's still quite a speedup.

Furthermore, all future improvements the AI makes to itself will now be amortized over 10,000 times as much computing power to apply the algorithms.  So a single improvement to code now has more impact than before; it's liable to produce more further improvements.  Think of a uranium pile.  It's always running the same "algorithm" with respect to neutrons causing fissions that produce further neutrons, but just piling on more uranium can cause it to go from subcritical to supercritical, as any given neutron has more uranium to travel through and a higher chance of causing future fissions.

So just the resource bonanza represented by "eating the Internet" or "discovering an application for which there is effectively unlimited demand, which lets you rent huge amounts of computing power while using only half of it to pay the bills" - even though this event isn't particularly recursive of itself, just an object-level fruit-taking - could potentially drive the AI from subcritical to supercritical.

Not, mind you, that this will happen with an AI that's just stupid.  But an AI already improving itself slowly - that's a different case.

Even if this doesn't happen - if the AI uses this newfound computing power at all effectively, its optimization efficiency will increase more quickly than before; just because the AI has more optimization power to apply to the task of increasing its own efficiency, thanks to the sudden bonanza of optimization resources.

So the whole trajectory can conceivably change, just from so simple and straightforward and unclever and uninteresting-seeming an act, as eating the Internet.  (Or renting a bigger cloud.)

Agriculture changed the course of human history by supporting a larger population - and that was just a question of having more humans around, not individual humans having a brain a hundred times as large.  This gets us into the whole issue of the returns on scaling individual brains not being anything like the returns on scaling the number of brains.  A big-brained human has around four times the cranial volume of a chimpanzee, but 4 chimps != 1 human.  (And for that matter, 60 squirrels != 1 chimp.)  Software improvements here almost certainly completely dominate hardware, of course.  But having a thousand scientists who collectively read all the papers in a field, and who talk to each other, is not like having one superscientist who has read all those papers and can correlate their contents directly using native cognitive processes of association, recognition, and abstraction.  Having more humans talking to each other using low-bandwidth words, cannot be expected to achieve returns similar to those from scaling component cognitive processes within a coherent cognitive system.

This, too, is an idiom outside human experience - we have to solve big problems using lots of humans, because there is no way to solve them using ONE BIG human.  But it never occurs to anyone to substitute four chimps for one human; and only a certain very foolish kind of boss thinks you can substitute ten programmers with one year of experience for one programmer with ten years of experience.

(Part of the general Culture of Chaos that praises emergence and thinks evolution is smarter than human designers, also has a mythology of groups being inherently superior to individuals.  But this is generally a matter of poor individual rationality, and various arcane group structures that are supposed to compensate; rather than an inherent fact about cognitive processes somehow scaling better when chopped up into distinct brains.  If that were literally more efficient, evolution would have designed humans to have four chimpanzee heads that argued with each other.  In the realm of AI, it seems much more straightforward to have a single cognitive process that lacks the emotional stubbornness to cling to its accustomed theories, and doesn't need to be argued out of it at gunpoint or replaced by a new generation of grad students.  I'm not going to delve into this in detail for now, just warn you to be suspicious of this particular creed of the Culture of Chaos; it's not like they actually observed the relative performance of a hundred humans versus one BIG mind with a brain fifty times human size.)

So yes, there was a lot of software improvement involved - what we are seeing with the modern human brain size, is probably not so much the brain volume required to support the software improvement, but rather the new evolutionary equilibrium for brain size given the improved software.

Even so - hominid brain size increased by a factor of five over the course of around five million years.  You might want to think very seriously about the contrast between that idiom, and a successful AI being able to expand onto five thousand times as much hardware over the course of five minutes - when you are pondering possible hard takeoffs, and whether the AI trajectory ought to look similar to human experience.

A subtler sort of hardware overhang, I suspect, is represented by modern CPUs have a 2GHz serial speed, in contrast to neurons that spike 100 times per second on a good day.  The "hundred-step rule" in computational neuroscience is a rule of thumb that any postulated neural algorithm which runs in realtime has to perform its job in less than 100 serial steps one after the other.  We do not understand how to efficiently use the computer hardware we have now, to do intelligent thinking.  But the much-vaunted "massive parallelism" of the human brain, is, I suspect, mostly cache lookups to make up for the sheer awkwardness of the brain's serial slowness - if your computer ran at 200Hz, you'd have to resort to all sorts of absurdly massive parallelism to get anything done in realtime.  I suspect that, if correctly designed, a midsize computer cluster would be able to get high-grade thinking done at a serial speed much faster than human, even if the total parallel computing power was less.

So that's another kind of overhang: because our computing hardware has run so far ahead of AI theory, we have incredibly fast computers we don't know how to use for thinking; getting AI right could produce a huge, discontinuous jolt, as the speed of high-grade thought on this planet suddenly dropped into computer time.

A still subtler kind of overhang would be represented by human failure to use our gathered experimental data efficiently.

On to the topic of insight, another potential source of discontinuity.  The course of hominid evolution was driven by evolution's neighborhood search; if the evolution of the brain accelerated to some degree, this was probably due to existing adaptations creating a greater number of possibilities for further adaptations.  (But it couldn't accelerate past a certain point, because evolution is limited in how much selection pressure it can apply - if someone succeeds in breeding due to adaptation A, that's less variance left over for whether or not they succeed in breeding due to adaptation B.)

But all this is searching the raw space of genes.  Human design intelligence, or sufficiently sophisticated AI design intelligence, isn't like that.  One might even be tempted to make up a completely different curve out of thin air - like, intelligence will take all the easy wins first, and then be left with only higher-hanging fruit, while increasing complexity will defeat the ability of the designer to make changes.  So where blind evolution accelerated, intelligent design will run into diminishing returns and grind to a halt.  And as long as you're making up fairy tales, you might as well further add that the law of diminishing returns will be exactly right, and have bumps and rough patches in exactly the right places, to produce a smooth gentle takeoff even after recursion and various hardware transitions are factored in...  One also wonders why the story about "intelligence taking easy wins first in designing brains" tops out at or before human-level brains, rather than going a long way beyond human before topping out.  But one suspects that if you tell that story, there's no point in inventing a law of diminishing returns to begin with.

(Ultimately, if the character of physical law is anything like our current laws of physics, there will be limits to what you can do on finite hardware, and limits to how much hardware you can assemble in finite time, but if they are very high limits relative to human brains, it doesn't affect the basic prediction of hard takeoff, "AI go FOOM".)

The main thing I'll venture into actually expecting from adding "insight" to the mix, is that there'll be a discontinuity at the point where the AI understands how to do AI theory, the same way that human researchers try to do AI theory.  An AI, to swallow its own optimization chain, must not just be able to rewrite its own source code; it must be able to, say, rewrite Artificial Intelligence: A Modern Approach (2nd Edition).  An ability like this seems (untrustworthily, but I don't know what else to trust) like it ought to appear at around the same time that the architecture is at the level of, or approaching the level of, being able to handle what humans handle - being no shallower than an actual human, whatever its inexperience in various domains.  It would produce further discontinuity at around that time.

In other words, when the AI becomes smart enough to do AI theory, that's when I expect it to fully swallow its own optimization chain and for the real FOOM to occur - though the AI might reach this point as part of a cascade that started at a more primitive level.

All these complications is why I don't believe we can really do any sort of math that will predict quantitatively the trajectory of a hard takeoff.  You can make up models, but real life is going to include all sorts of discrete jumps, bottlenecks, bonanzas, insights - and the "fold the curve in on itself" paradigm of recursion is going to amplify even small roughnesses in the trajectory.

So I stick to qualitative predictions.  "AI go FOOM".

Tomorrow I hope to tackle locality, and a bestiary of some possible qualitative trajectories the AI might take given this analysis.  Robin Hanson's summary of "primitive AI fooms to sophisticated AI" doesn't fully represent my views - that's just one entry in the bestiary, albeit a major one.