For some odd reason, I run into a lot of people who vigorously deny that this phenomenon is at all novel; they say, "Oh, humanity is already self-improving, humanity is already going through a FOOM, humanity is already in a Singularity" etc. etc.

People like me. I don't see much of a counter-argument in this post - or at least not a coherent one.

The future contains a feedback loop that the past does not.

It seems like a denial of sexual selection to me. Brainpower went into making new brains historically - via sexual selection. Feedback from the previous generation of brains into the next generation has taken place historically.

It's also a denial of cultural evolution - which is even more obvious. The past contains several decades of Moore's law - the product of a self-improving computer industry. It seems impossible to deny that computers help to design the next generation of computers. They are doing more of the work in each passing generation.

Feedback from the current generation of thinking equipment into the next one is an ancient phenomenon. Even directed mutations and intelligent design are quite old news now.

One of the few remaining dramatic future developments will occur when humans drop out of the loop - allowing it to iterate at more like full speed.

the initial recursive cascade of an intelligence explosion can't race through human brains because human brains are not modifiable until the AI is already superintelligent

I am extremely sceptical about whether we will see much modification of human brains by superintelligent agents. Once we have superintelligence, human brains will go out of fashion the way the horse-and-cart did.

Brains will not suddenly become an attractive platform for future development with the advent of superintelligence - rather they will become even more evidently obsolete junk.

The problem, as I see it, is that you can't take bits out of a running piece of software and replace them with other bits, and have them still work, unless said piece of software is trivial.

The capacity to do in-place updates of running software components dates back to at least the first LISP systems. Call it 1955? Modern day telephone switches and network routers are all built with the capability of doing hot upgrades, or they wouldn't be able to reach the the level of uptime required (if you require 99.9999% uptime, going down for 30 seconds for an upgrade ruins your numbers for ten years). Additionally, those systems require that every component be independently crashable and restartable, for reliability purposes.

The problem, as I see it, is that you can't take bits out of a running piece of software and replace them with other bits, and have them still work, unless said piece of software is trivial.

The AI could probably do a reboot if it needed to. For that matter, the computer you're writing this on probably has modular device drivers that can be replaced without the machine needing a reboot.

"Recursion that can rewrite the cognitive level is worth distinguishing."

Eliezer, would a human that modifies the genes that control how his brain is built qualify as the same class of recursion (but with a longer cycle-time), or is it not quite the same?

Andrew, we're not talking about the equivalent of a human studying neuroscience by groping in the dark. If an AI truly, truly groks every line of its own code, it can pretty much do what it wants with it. No need for trial and error when you have full visibility and understanding of every neuron in your head.

How, you ask? What do such recursive algorithms look like? Mere details; the code monkeys can worry about all that stuff!

Inventing science is not rewriting your neural circuitry. [...] To see the brain circuitry vary, you've got to look at a chimpanzee, basically.

You should be considering augmented humans as single cybernetic entities when you write like this.

Augmented humans are a bunch of sensors linked to a bunch of actuators via a load of computing power - it's just that some of the processing is done by machinery, not neurons.

Then there is substantial variation in the capabilities of the resulting entities - depending on to what extent they augment themselves.

Just looking at the bit with the neurons totally misses out the section where all the action and change is taking place!

To then claim that there's no self-improvement action happening yet is broadly correct - from that blinkered perspective.

However, the reality is that humans don't think with just their bare brains. They soak them in culture and then augment them with machinery. Consider those proceses, and you should see the self-improvement that has taken place so far.

I think you have a tendency to overlook our lack of knowledge of how the brain works. You talk of constant brain circuitry, when people add new hippocampal cells through their life. We also expand the brain areas devoted to fingers if we are born blind and use braille.

We don't know how else the brain rewires itself. In some sense all knowledge is wiring in the brain... I mean what else is it going to be. This is all invisible to us, and may throw a spanner in the works of any intelligence trying to improve a moving target.

Depending on which abstractions you emphasize, you can describe a new thing as something completely new under the sun, or as yet another example of something familiar. So the issue is which abstractions make the most sense to use. We have seen cases before where when one growth via some growth channel opened up more growth channels, to further enable growth. So the question is how similar those situations are to this situation, where and AI getting smarter allows an AI to change its architecture in more and better ways. Which is another way of asking which abstractions are most relevant.

The rapidity of evolution from chimp to human is remarkable, but you can infer what you're trying to infer only if you believe evolution reliably produces steadily more intelligent creatures. It might be that conditions temporarily favored intelligence, leading to humans; our rapid rise is then explained by the anthropic principle, not by universal evolutionary dynamics.

Knowledge = all that actual science, engineering, and general knowledge accumulation we did = integral of cognition+metaknowledge(current knowledge) over time, where knowledge feeds upon itself in what seems to be a roughly exponential process

Knowledge feeds on itself only when it is continually spread out over new domains. If you keep trying to learn more about the same domain - say, to cure cancer, or make faster computer chips - you get logarithmic returns, requiring an exponential increase in resources to maintain constant output. (IIRC it has required exponentially-increasing capital investments to keep Moore's Law going; the money will run out before the science does.) Rescher wrote about this in the 1970s and 1980s.

This is important because it says that, if an AI keeps trying to learn how to improve itself, it will get only logarithmic returns.

When you fold a complicated, choppy, cascade-y chain of differential equations in on itself via recursion, it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly through the extremely narrow soft takeoff keyhole.

This is the most important and controversial claim, so I'd like to see it better-supported. I understand the intuition; but it is convincing as an intuition only if you suppose there are no negative feedback mechanisms anywhere in the whole process, which seems unlikely.

I think it will be a more intuitive abstraction to talk about an AI that designs the next (at the object level) AI using itself as a template. None of us has conceptual experience with directly modifying our own brains. All of us have built something. We also have good natural language tools to deal with things like 'grandchildren' and 'inheriting the knowledge base'. Modifying code in place would be a more likely reality, but I don't think it's relevant to the argument, and just makes the whole thing harder to think about.

Yes we're talking about a self modifying AI design, but each iteration is in many ways a new entity.

This also puts the sexual selection as brains designing brains into perspective. A proto-woman could only choose among the men available to her, which were the result of natural selection "searching the immediate neighborhood of" the previous generation. She can nudge evolution, but she can't use her insight to make a substantially better mate. ( I think this is 'metacognitive' vs. 'object level'

Even to the degree that there was a feedback loop, it didn't just have 'smart' as its goal. Brains designing brains doesn't get you foom if they're designing/selecting for 'funny' or 'apparently faithful but not really'.

"If computing power doubles every eighteen months, what happens when computers are doing the research?"

And this sounds horrifyingly naive to my present ears

TMOL was freaking brilliant. This post was awesome. It blew me away. Can't wait to see the follow up.

I know this whole comment was kind of vacuous, but yeah. Wow. I don't even have to tell you that you make more sense than virtually everyone on the planet.

http://yudkowsky.net/obsolete/tmol-faq.html

I'd like to focus on the example offered: "Write a better algorithm than X for storing, associating to, and retrieving memories." Is this a well defined task?

Wouldn't we want to ask, better by what measure? Is there some well defined metric for this task? And what if it is better at storing, but worse at retrieving, or vice versa? What if it gets better quality answers but takes longer - is that an improvement or not? And what if the AI looks at the algorithm, works on it for a while, and admits in the end that it can't see a way to really make it better? I'm a professional software writer and there are not many standard algorithms where I'd expect to be able to come up with a significant improvement.

As I puzzle over these issues, I am confused whether this example is supposed to represent a task where we (or really, the AI, for I think we are to imagine that it is setting itself the task) would know exactly how to quantify "better"? Or is this associating-memory functionality supposed to be some of the deep dark hidden algorithmic depths of our mind, and the hard part will be figuring out how AI-enabling associative memory works in the first place? And then once we have done so, would improving it be a basically mechanical task which any moderately smart AI or human would certainly be able to accomplish? Or would it require a super-genius AI which can figure out improvements on almost any human-designed algorithm?

And does this example really lead to a recursive cycle of self-improvement, or is it like the optimizing compiler, which can speed up its database access operations but that doesn't make it fundamentally any smarter?

From a practical point of view, a "hard takeoff" would seem to be defined by self-improvement and expansion of control at a rate too fast for humans to cope with. As an example of this, it is often put forward as obvious that the AI would invent molecular nanotechnology in a matter of hours.

Yet there is no reason to think it's even possible to improve molecular simulation, required to search in molecular process-space, much beyond our current algorithms, which on any near-term hardware are nowhere near up to the task. The only explanation is that you are hypothesizing rather incredible increases in abilities such as this without any reason to even think that they are possible.

It's this sort of leap that makes the scenario difficult to believe. Too many miracles seem necessary.

Personally I can entertain the possibility of a "takeoff" (though it is no sure thing that it is possible), but the level of optimization required for a hard takeoff seems unreasonable. It is a lengthy process just to compile a large software project (a trivial transformation). There are limits to what a particular computer can do.

Hal: At some point, you've improved a computer program. You had to decide, somehow, what tradeoffs to make, on your own. We should assume that a superhuman AI will be at least as good at improving programs as we are.

I can't think of any programs of broad scope that I would call unimprovable. (The AI might not be able to improve this algorithm this iteration, but if it really declares itself perfectly optimized, I'd expect we would declare it broken. In fact that sounds like the EURISKO. An AGI should a least keep trying.)

Also: Any process that it knows how to do, that it has learned, it can implement in its own code, so it does not have to 'think things out' with it's high-level thinking algorithms. This is repeatable for everything it learns. (We can't do the same thing to create an AI because we don't have access to our algorithms or really even our memories. If an AI can learn to recognize breeds of dogs, then it can trace its own thoughts to determine by what process it does that. Since the learning algorithm probably isn't perfectly optimized to learn how to recognize dogs, the learned process it is using is probably not perfectly efficient.)

The metacognitive level becoming part of the object level lets you turn knowledge and metaknowledge directly into cognitive improvements. For every piece of knowledge, including knowledge about how to program.

Phil: Anthropic pressures should by default be expected to be spread uniformly through our evolutionary history accelerating the evolutionary and pre-evolutionary record of events leading to us rather than merely accelerating the last stretch.

Exponential inputs into computer chip manufacture seem to produce exponential returns with a doubling time significantly less than that for the inputs, implying increasing returns per unit input, at least if one measures in terms of feature number. Obviously returns are exponentially diminishing if one measures in time to finish some particular calculation. Returns will more interestingly be diminishing per unit labor in terms of hardware design effor per unit of depth to which a NP and exponential complexity class problems can be calculated, e.g. the number of moves ahead a chess program can look. OTOH, it bizarrely appears to be the case that over a large range of chess ranks, human players seem to gain effective chess skill measured by chess rank with roughly linear training while chess programs gain it via exponential speed-up.

Society seems to in aggregate get constant zero returns on efforts to cure cancer, though one can't rule out exponential returns starting from zero. OTOH, this seems consistent with the general inefficacy of medicine in aggregate as shown by the Rand study, which doesn't overturn the individual impacts, as shown by FDA testing, of many individual medical procedures. Life expectancy in the US has grown linearly while GDP per capita has grown exponentially, but among nations in the modern world life expectancy clearly has a different relationship to income, not linear, not logarithmic, more plausibly asymptotic moving towards something in the early 80s.

I'm glad that you consider the claim about turning object level knowledge metacognitive to be the most important and controvercial claim. This seems like a much more substantial and precise criticism of Eliezer's position than anything Robin has made so far. It would be very interesting to see you and Eliezer discuss evidence for or against sufficient negative feedback mechanisms, Eliezer's "just the right law of diminishing retunrs" existing.

James Andrix: In fact that sounds like the EURISKO.

Could you elaborate? My understanding is that Eurisko never gave up, but Lenat got bored of babysitting it.

(Compound reply from Eliezer.)

Eliezer: When you fold a complicated, choppy, cascade-y chain of differential equations in on itself via recursion, it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly through the extremely narrow soft takeoff keyhole.

Goetz: This is the most important and controversial claim, so I'd like to see it better-supported. I understand the intuition; but it is convincing as an intuition only if you suppose there are no negative feedback mechanisms anywhere in the whole process, which seems unlikely.

Can you give a plausible example of a negative feedback mechanism as such, apart from a law of diminishing returns that would be (nearly) ruled out by historical evidence already available?

I suspect that human economic growth would naturally tend to be faster and somewhat more superexponential, if it were not for the negative feedback mechanism of governments and bureaucracies with poor incentives, that both expand and hinder whenever times are sufficiently good that no one is objecting strongly enough to stop it; when "economic growth" is not the issue of top concern to everyone, all sorts of actions will be taken to hinder economic growth; when the company is not in immediate danger of collapsing, the bureaucracies will add on paperwork; and universities just go on adding paperwork indefinitely. So there are negative feedback mechanisms built into the human economic growth curve, but an AI wouldn't have them because they basically derive from us being stupid and having conflicting incentives.

What would be a plausible negative feedback mechanism - as apart from a law of diminishing returns? Why wouldn't the AI just stomp on the mechanism?

Hanson: Depending on which abstractions you emphasize, you can describe a new thing as something completely new under the sun, or as yet another example of something familiar. So the issue is which abstractions make the most sense to use. We have seen cases before where when one growth via some growth channel opened up more growth channels, to further enable growth. So the question is how similar those situations are to this situation, where an AI getting smarter allows an AI to change its architecture in more and better ways. Which is another way of asking which abstractions are most relevant.

Well, the whole post above is just putting specific details on that old claim, "Natural selection producing humans and humans producing technology can't be extrapolated to an AI insightfully modifying its low-level brain algorithms, because the latter case contains a feedback loop of an importantly different type; it's like trying to extrapolate a bird flying outside the atmosphere or extrapolating the temperature/compression law of a gas past the point where the gas becomes a black hole."

If you just pick an abstraction that isn't detailed enough to talk about the putative feedback loop, and then insist on extrapolating out the old trends from the absence of the feedback loop, I would consider this a weak response.

Pearson: I think you have a tendency to overlook our lack of knowledge of how the brain works. You talk of constant brain circuitry, when people add new hippocampal cells through their life. We also expand the brain areas devoted to fingers if we are born blind and use braille.

Pearson, "constant brains" means "brains with constant adaptation-algorithms, such as an adaptation-algorithm for rewiring via reinforcement" not "brains with constant synaptic networks". I think a bit of interpretive charity would have been in order here.

Finney: I'd like to focus on the example offered: "Write a better algorithm than X for storing, associating to, and retrieving memories." Is this a well defined task? Wouldn't we want to ask, better by what measure? Is there some well defined metric for this task?

Hal, if this is taking place inside a reasonably sophisticated Friendly AI, then I'd expect there to be something akin to an internal economy of the AI with expected utilons as the common unit of currency. So if the memory system is getting any computer time at all, the AI has beliefs about why it is good to remember things and what other cognitive tasks memory can contribute to. It's not just starting with an inscrutable piece of code that has no known purpose, and trying to "improve" it; it has an idea of what kind of labor the code is performing, and which other cognitive tasks that labor contributes to, and why. In the absence of such insight, it would indeed be more difficult for the AI to rewrite itself, and its development at that time would probably be dominated by human programmers pushing it along.

Ian C.: Eliezer, would a human that modifies the genes that control how his brain is built qualify as the same class of recursion (but with a longer cycle-time), or is it not quite the same?

Owing to our tremendous lack of insight into how genes affect brains, and owing to the messiness of the brain itself as a starting point, we would get relatively slow returns out of this kind of recursion even before taking into account the 18-year cycle time for the kids to grow up.

However, on a scale of returns from ordinary investment, the effect on society of the next generation being born with an average IQ of 140 (on the current scale) might be well-nigh inconceivable. It wouldn't be an intelligence explosion; it wouldn't be the kind of feedback loop I'm talking about - but as humans measure hugeness, it would be huge.

Reid: I'm sure you're aware of Schmidhuber's forays into this area with his Gödel Machine. Doesn't this blur the boundaries between the meta-cognitive and cognitive?

Schmidhuber's "Gödel Machine" is talking about a genuine recursion from object-level to metacognitive level, of the sort I described. However, this problem is somewhat more difficult than Schmidhuber seems to think it is, to put it mildly - but that would be part of the AIXI sequence, which I don't think I'll end up writing. Also, I think some of Schmidhuber's suggestions potentially hamper the system with a protected level.

Vassar: OTOH, it bizarrely appears to be the case that over a large range of chess ranks, human players seem to gain effective chess skill measured by chess rank with roughly linear training while chess programs gain it via exponential speed-up.

I expect that what you're looking at is a navigable search space that the humans are navigating and the AI is grasping through brute-force techniques - yes, Deep Blue wasn't literally brute force, but it was still navigating raw Chess rather than Regularity in Chess. If you're searching the raw tree, returns are logarithmic; the human process of grokking regularities seems to deliver linear returns over practice with a brain in good condition. However, with Moore's Law in play (exponential improvements delivered by human engineers) the AIs outran the brains.

Humans getting linear returns where dumb algorithms get logarithmic returns, seems to be a fairly standard phenomenon in my view - consider natural selection trying to go over a hump of required simultaneous changes, for example.

Tim Tyler: Brainpower went into making new brains historically - via sexual selection. Feedback from the previous generation of brains into the next generation has taken place historically.

If no one besides me thinks this claim is credible, I'll just go ahead and hold it up as an example of the kind of silliness I'm talking about, so that no one accuses me of attacking a strawman.

(Quick reductio: Imagine Jane Cavewoman falling in love with Johnny Caveman on the basis of a foresightful extrapolation of how Johnny's slightly mutated visual cortex, though not useful in its own right, will open up the way for further useful mutations, thus averting the unforesightful basis of natural selection... Sexual selection just applies greater selection pressure to particular characteristics; it doesn't change the stupid parts of evolution at all - in fact, it often makes evolution even more stupid by decoupling fitness from characteristics we would ordinarily think of as "fit" - and this is true even though brains are involved. Missing this and saying triumphantly, "See? We're recursive!" is an example of the overeager rush to apply nice labels that I was talking about earlier.)

Drucker: The problem, as I see it, is that you can't take bits out of a running piece of software and replace them with other bits, and have them still work, unless said piece of software is trivial... The human brain is a mass of interconnecting systems, all tied together in a mish-mash of complexity. You couldn't upgrade any one part of it by finding a faster replacement for any one section of it. Attempting to perform brain surgery on yourself is going to be a slow, painstaking process, leaving you with far more dead AIs than live ones.

As other commenters pointed out, plenty of software is written to enable modular upgrades. An AI with insight into its own algorithms and thought processes is not making changes by random testing like it was bloody evolution or something. A Friendly AI uses deterministic abstract reasoning in this case - I guess I'd have to write a post about how that works to make the point, though.

A poorly written AI might start out as the kind of mess you're describing, and of course, also lack the insight to make changes better than random; and in that case, would get much less mileage out of self-improvement, and probably stay inert.

Douglas: I was going by Eliezer's description. It wasn't smart enough to improve itself anymore [quickly?]. If nothing else, a Human level AI should be able to think more about the problem for a while, profile its thought process, and hardcode that process.

Eliezer: part of the AIXI sequence, which I don't think I'll end up writing.

Ahh, that's a shame, though fully understood. Don't suppose you (or anyone) can link to some literature about AIXI? Haven't been able to find anything comprehensive yet comprehensible to an amateur.

Tim Tyler: Brainpower went into making new brains historically - via sexual selection. Feedback from the previous generation of brains into the next generation has taken place historically.

Tim, Dawkins has a nice sequence in The Blind Watchmaker about a species of bird in which the female began selecting for big, lustrous tails. This led to birds with tails so big they could barely fly to escape predators. While selecting for intelligence in a partner is obviously plausible, I'd have to see very compelling evidence that it's leading to continuously smarter people, or even that it ever could. Possibly a loop of some description there, but K definitely < 1.

However!

Owing to our tremendous lack of insight into how genes affect brains

So what happens when we start to figure out what those genes do? and then start switching them on and off, and gaining more knowledge and insight into how the brain attacks problems? As we've read recently, natural selection increased brainpower (insight) massively through blind stumbling onto low-hanging fruit in a relatively small amount of time. Why would we suppose it reached any sort of limit - or at least a limit we couldn't surmount? The 18 years to maturity thing is pretty irrelevant here, as long as, say, 5% compound insight can be gained per generation. You're still looking at exponential increases, and you might only need a handful of generations before the FOOM itself switches medium.

Ben, that would indeed be the path of humanity's future, I expect, if not for those pesky computers - and the possibility of nanotech and uploading - and all the other interesting things that would happen long before another ten cycles of the flesh.

"I suspect that human economic growth would naturally tend to be faster and somewhat more superexponential, if it were not for the negative feedback mechanism of governments and bureaucracies with poor incentives, that both expand and hinder whenever times are sufficiently good that no one is objecting strongly enough to stop it; when "economic growth" is not the issue of top concern to everyone, all sorts of actions will be taken to hinder economic growth; when the company is not in immediate danger of collapsing, the bureaucracies will add on paperwork; and universities just go on adding paperwork indefinitely."

You may consider expanding this into a post - it is one of the most important insights for the shorter term future I have yet seen on OB (or most other blogs).

A number of people are objecting to Eliezer's claim that the process he is discussing is unique in its FOOM potential, proposing other processes that are similar. Then Eliezer says they aren't similar.

Whether they're similar enough depends on the analysis you want to do. If you want to glance at them and come up with yes or no answer regarding FOOM, then none of them are similar. A key difference is that these other things don't have continual halving of the time per generation. You can account for this when comparing results, but I haven't seen anyone do this.

But some things are similar enough that you can gain some insights into the AI FOOM potential by looking at them. Consider the growth of human societies. A human culture/civilization/government produces ideas, values, and resources used to rewrite itself. This is similar to the AI FOOM dynamics, except with constant and long generation times.

To a tribesman contemplating the forthcoming culture FOOM, it would look pretty simple: Culture is about ways for your tribe to get more land than other tribes.

As culture progressed, we developed all sorts of new goals for it that the tribesman couldn't have predicted.

Analogously, our discussion of the AI FOOM supposes that the AI will not discover new avenues to pursue other than intelligence, that soak up enough of the FOOM to slow down the intelligence part of the FOOM considerably. (Further analysis of this is difficult since we haven't agreed what "intelligence" is.)

Another lesson to learn from culture has to do with complexity. The tribesman, given some ideas of what technology and government would do, would suppose that it would solve all problems. But in fact, as cultures grow more capable, they are able to sustain more complexity; and so our problems get more and more complicated. The idea that human stupidity is holding us back, and AIs will burst into exponential territory once they shake free of these shackles:

I suspect that human economic growth would naturally tend to be faster and somewhat more superexponential, if it were not for the negative feedback mechanism of governments and bureaucracies with poor incentives, that both expand and hinder whenever times are sufficiently good that no one is objecting strongly enough to stop it

is like that tribesman thinking good government will solve all problems. Systems - societies, governments, AIs - expand to the limits of complexity that they can support; at those limits, actions have unintended consequences and agents have not quite enough intelligence to predict them or agree on them, and in efficiency and "stupidity" - relative stupidity - lives on.

I'll respond to Eliezer's response to my response later today. Short answer: 1. Diminishing returns exist and are powerful. 2. This isn't something you can eyeball. If you want to say FOOM is probable; fine. If you want to say FOOM is almost inevitable, I want to see equations worked out with specific numbers. You won't convince me with handwaving, especially when other smart people are waving their hands and reaching different conclusions.

I wrote:

Analogously, our discussion of the AI FOOM supposes that the AI will not discover new avenues to pursue other than intelligence, that soak up enough of the FOOM to slow down the intelligence part of the FOOM considerably.

What I wish I'd said is: What percentage of the AI's efforts will go into algorithm, architecture, and hardware research?

At the start, probably a lot; so this issue may not be important wrt FOOM and humans.

One source of diminishing returns is upper limits on what is achievable. For instance, Shannon proved that there is an upper bound on the error-free communicative capacity of a channel. No amount of intelligence can squeeze more error-free capacity out of a channel than this. There are also limits on what is learnable using just induction, even with unlimited resources and unlimited time (cf "The Logic of Reliable Inquiry" by Kevin T. Kelly). These sorts of limits indicate that an AI cannot improve its meta-cognition exponentially forever. At some point, the improvements have to level off.

Sexual selection is at the root of practically all the explanations for the origin of our large brains. To quote from Sue Blackmore:

An influential version of social theory is the ‘Machiavellian Intelligence’ hypothesis (Byrne and Whiten 1988; Whiten and Byrne 1997). Social interactions and relationships are not only complex but also constantly changing and therefore require fast parallel processing (Barton and Dunbar 1997). The similarity with Niccolò Machiavelli (1469–1527), the devious adviser of sixteenth-century Italian princes, is that much of social life is a question of outwitting others, plotting and scheming, entering into alliances and breaking them again. All this requires a lot of brain power to remember who is who, and who has done what to whom, as well as to think up ever more crafty wiles, and to double bluff the crafty wiles of your rivals – leading to a spiralling arms race. ‘Arms races’ are common in biology, as when predators evolve to run ever faster to catch their faster prey, or parasites evolve to outwit the immune systems of their hosts. The notion that some kind of spiralling or self-catalytic process is involved certainly suits what Christopher Wills (1993) calls ‘the runaway brain’, and this idea is common among theories that relate language evolution to brain size.

The idea that intelligence can play an important role in evolutionary change arises from the observation that intelligent agents are doing the selecting - in sexual selection. They get to use induction, deduction, analogies, prediction - the whole toolkit of intelligence - and the results are then reflected in the germ line of the next generation. Any idea that the loop between intelligence and brain design information has only closed recently - or has yet to close - is simply wrong. Human brains have been influencing human brain evolution for millions of years - in the same way that they have been influencing dog evolution for millions of years - by acting as the selective agent.

Brains don't just choose. They also create circumstances where there is a lot of information on which to make choices. The mating dance brainy-females often lead males on exposes their parasite load - and so their genetic quality - to selection. So: brains do not just select - they actively create opportunties for selection by intelligence to have a large influence.

If you look at female secondary sex charateristics, they are the products of mind, written back onto the body by evolution. Breasts, red lips - and so on are physical projections of mental analogy-making equipment.

Of course the other classic example of the mental turning into the genetic is the Baldwin effect - where learned, acquired characteristics find their way out of the minds in which they arose and back into the gene pool.

Evolution is no stranger to the action of intelligence - indeed, without millions of intelligent choices by our ancestors, the human race as we know it today would not exist.

John: Given any universe whose physics even resembles in character our current standard model, there will be limits to what you can do on fixed hardware and limits to how much hardware you can create in finite time.

But if those limits are far, far, far above the world we think of as normal, then I would consider the AI-go-FOOM prediction to have been confirmed. I.e., if an AI builds its own nanotech and runs off to disassemble Proxima Centauri, that is not infinite power but it is a whole lot of power and worthy of the name "superintelligence".

If you want to say FOOM is probable; fine. If you want to say FOOM is almost inevitable, I want to see equations worked out with specific numbers.

The explosion in computing capability is a historical phenomenon that has been going on for decades. For "specific numbers", for example, look at the well-documented growth of the computer industry since the 1950s. Yes, there are probably limits, but they seem far away - so far away, we are not even sure where they are, or even whether they exist.

"The explosion in computing capability is a historical phenomenon that has been going on for decades. For "specific numbers", for example, look at the well-documented growth of the computer industry since the 1950s. Yes, there are probably limits, but they seem far away - so far away, we are not even sure where they are, or even whether they exist."

The growth you are referring to has a hard upper limit which is when transistors are measured in angstroms, at the point when they start playing by the rules of quantum mechanics. That is the hard upper limit of computing that you are referring to. Now if we take quantum computing that may or may not take us further there has been a lot of work done recently that casts doubt on quantum computing and its ability to solve a lot of our computing issues. There are a lot of other possible computing technologies it is just not clear which one will emerge at the top yet.

Pearson: I think you have a tendency to overlook our lack of knowledge of how the brain works. You talk of constant brain circuitry, when people add new hippocampal cells through their life. We also expand the brain areas devoted to fingers if we are born blind and use braille.

Pearson, "constant brains" means "brains with constant adaptation-algorithms, such as an adaptation-algorithm for rewiring via reinforcement" not "brains with constant synaptic networks". I think a bit of interpretive charity would have been in order here. We don't know how deep the rabbit hole of adaptation goes. Are there constant adaptation-algortihms? Constant adaptation algorithms are not a prerequisite for an optimization process, evolution being the cannonical example. It gets by with a very changeable adaptation-algorithm embodied in the varieties of genetic transfer, reproduction rates etc. We have barely scratched the surface of adaption systems, assuming a constant adaptation-algorithm for intelligence is premature, as far as I am concerned.

Eliezer: If "AI goes FOOM" means that the AI achieves super-intelligence within a few weeks or hours, then it has to be at the meta-cognitive level or the resource-overhang level (taking over all existing computer cycles). You can't run off to Proxima Centauri in that time frame.

"For "specific numbers", for example, look at the well-documented growth of the computer industry since the 1950s."

You would need to show how to interpret those numbers applied to the AI foom.

I'd rather see a model for AI foom built from the ground up, and ranges of reasonable values posited, and validated in some way.

This is a lot of work, but after several years working on the problem, it's one that ought to have a preliminary answer.

Increasingly powerful machines result in increasingly powerful machine intelligence. We built most of those machines to augment human abilities, including - prominently - human intelligence. The faster they get the smarter they are - since one component of intelligence is speed.

" The faster they get the smarter they are - since one component of intelligence is speed."

I think this might be incorrect. The speed means that you can solve the problem faster, not that you can solve more complex problem.

Intelligence tests are timed for a good reason. If you see intelligence as an optimisation process, it is obvious why speed matters - you can do more trials. There are some cases where it's optimisation / trials that's important - but plenty where it's just getting the answer quickly that matters.

Tim:

It seems to me you are being almost deliberately obtuse. Of course the brain has developed over the long course of evolution via sexual selection. The same process happens in parakeets. They have brains because they evolved brains. Some of that brain power went to making them better at catching prey, and some went to making them better mating-call singers, leading to larger brain size in the next generation. Humans just happen to be the brainiest of all creatures; but the mechanism is the same. You might as well argue that the "intelligence explosion" starts with the first prokaryote.

Ditto your argument about machines being used to improve machines. You know, cave dwellers used tools to improve their tools. Is that when the "intelligence explosion" began with recursion?

You are not even talking about the same topic that Eliezer is. I'm a low-ranking grunt in the rationalist army, and usually I just lurk here because I feel often feel out of my depth. But it's frustrating to see the thread get jacked by an argument that has no real content.

The problem you are having is that all of your individual points are true as far as they go, but you're just restating agreed-upon facts using some borrowed terms to make it sound grand and important. I suggest you re-read the posts Eliezer linked to about the "virtue of narrowness" and "sounding wise versus being wise."

Now, back to my lurker's cave.

"Intelligence tests are timed for a good reason. If you see intelligence as an optimisation process, it is obvious why speed matters - you can do more trials."

Inteligence tests are designed to measure performance of human brain.

Try this: Strong AI running on 2Ghz CPU. You reduce it to 1Ghz, without changing anything else. Will it make less intelligent? Slower, definitely.

Of course the brain has developed over the long course of evolution via sexual selection.

This is not obvious - and indeed for a long time it was not even a popular theory. Many people thought tool use - and its resulting survival characteristics was the important factor:

"Most traditional theories, including that of Charles Darwin, suggested some combination of tool use and hunting were the key selective pressures favoring big brains, but increasing evidence of hunting and tool use in other species such as chimpanzees indicates our ancestors were not unique in that regard," Flinn said. "The most exceptional of our mental gifts involves understanding what is going on in other people's minds by using skills such as empathy and self-awareness.

Still today, one viable theory is that the human brain developed as a result of nutritional constraints being lifted - as a result of a diet including meat and seafood providing omega-rich fatty acids in abundance (see "The Driving Force") - and that theory makes little reference to sexual selection.

You might as well argue that the "intelligence explosion" starts with the first prokaryote.

I refer to that as the "technology explosion" (though strictly that began much earlier). The term "intelligence" is usually associated with organisms which have brains - and I do indeed argue that the explosion began with the origin of brains.

You know, cave dwellers used tools to improve their tools. Is that when the "intelligence explosion" began with recursion?

In my view, the intelligence explosion is best though of as beginning with the origin of animal brains - since that is when we have evidence that brain size began increasing exponentially. So the answer to your question is "no": the intelligence explosion did not begin with cave dwellers. It is curious that you would ask such a wrong question if you had read my essay on the subject.

I am not the one using the "recursion" terminology, without saying clearly what is doing the recursing. My point is if what is doing the recursing is "intelligence" then this is not a new trick for evolution - far from it. Intelligence has been intimately involved in the origin of the next generation for millions of years. So it needs to be specified what type of recursion is going on - else the whole point falls flat.

What is going to be new in the future? It's not intelligence - but it is intelligent design. Previously we only really had intelligent selection. Intelligent mutations were there too in principle - but they had to be time-consumingly transferred into the germ-line via the Baldwin effect.

Eliezer, your basic error regarding the singularity is the planning fallacy. And a lot of people are going to say "I told you so" sooner or later.

Unknown, it can't literally be a planning fallacy because I'm not giving either a plan (a series of exact events) or a timeframe. You perhaps meant to say "conjunction fallacy"? If so, you're still wrong. Listing a dozen different ways that things can exhibit sharp jumps is a disjunction, not a conjunction. "Too far" is a powerful accusation, but I don't think that either "planning fallacy" or "conjunction fallacy" should qualify here.

Because it shows that with constant optimization pressure from natural selection and no intelligent insight, there were no diminishing returns to a search for better brain designs up to at least the human level. There were probably accelerating returns (with a low acceleration factor). There are no visible speedbumps, so far as I know.

Were the brain designs better because they were more powerful or more intelligent?

That is how many of the improvements were adding more resources to the brain (because they paid off in this evolutionary case), rather than adding more efficient programs/systems.

it's as simple as identifying "recursive self-improvement" as a term with positive affective valence, so you figure out a way to apply that term to humanity, and then you get a nice dose of warm fuzzies.

recursive self-improvement is possible and has been effortlessly achieved by many people.

"Intelligence tests are timed for a good reason. If you see intelligence as an optimisation process, it is obvious why speed matters - you can do more trials."

Regarding the differential equations math: Isn't this pretty much exactly what Chalmers' "proportionality thesis" formalizes into? And additionally, since m is in the exponent, wouldn't that make 2 AI teams working with different amounts of hardware like investors investing their money at different interest rates?