I've argued that we might have to worry about dangerous non-general intelligences. In a series of back and forth with Wei Dai, we agreed that some level of general intelligence (such as that humans seem to possess) seemed to be a great advantage, though possibly one with diminishing returns. Therefore a dangerous AI could be one with great narrow intelligence in one area, and a little bit of general intelligence in others.

The traditional view of an intelligence explosion is that of an AI that knows how to do X, suddenly getting (much) better at doing X, to a level beyond human capacity. Call this the gain of aptitude intelligence explosion. We can prepare for that, maybe, by tracking the AI's ability level and seeing if it shoots up.

But the example above hints at another kind of potentially dangerous intelligence explosion. That of a very intelligent but narrow AI that suddenly gains intelligence across other domains. Call this the gain of function intelligence explosion. If we're not looking specifically for it, it may not trigger any warnings - the AI might still be dumber than the average human in other domains. But this might be enough, when combined with its narrow superintelligence, to make it deadly. We can't ignore the toaster that starts babbling.

I've created a new website for my ebook Facing the Intelligence Explosion:

 

Sometime this century, machines will surpass human levels of intelligence and ability, and the human era will be over. This will be the most important event in Earth’s history, and navigating it wisely may be the most important thing we can ever do.

Luminaries from Alan Turing and Jack Good to Bill Joy and Stephen Hawking have warned us about this. Why do I think they’re right, and what can we do about it?

Facing the Intelligence Explosion is my attempt to answer those questions.

 

This page is the dedicated discussion page for Facing the Intelligence Explosion.

If you'd like to comment on a particular chapter, please give the chapter name at top of your comment so that others can more easily understand your comment. For example:

Re: From Skepticism to Technical Rationality

Here, Luke neglects to mention that...

Again, I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on. This section is less complete than the others; missing text is indicated with brackets: [].

_____

From digital intelligence to intelligence explosion

Humans are the first terrestrial intelligences sophisticated enough to produce a technological civilization. It seems unlikely that we are near the ceiling of possible intelligences, rather than simply being the first such intelligence that happened to evolve. Computers far outperform humans in many narrow niches, and there is reason to believe that similar large improvements over human performance are possible for general reasoning, technology design, and other tasks of interest.

Below, we discuss the advantages of digital intelligence that will likely produce an intelligence explosion, along with the nature and speed of this “takeoff.”

Advantages from mere digitality

One might think the first human-level digital intelligence would not affect the world much. After all, we have seven billion human-level intelligences right now, and few or none of them have prompted massive, sudden AI breakthroughs. A single added intelligence might seem to be a tiny portion of the total drivers of technological innovation.

But while human-level humans do not suddenly lead to smarter AI, there are reasons to expect human-level digital intelligence to enable faster change. Digital intelligences can be ported across hardware platforms, and this has at least three significant consequences: speed, copyability, and goal coordination.

Speed

Axons carry spike signals at 75 meters per second or less (Kandel et al. 2000). That speed is a fixed consequence of the type of physiology we humans run on. In contrast, software minds could be ported to any available hardware, and could therefore think more rapidly as faster hardware became available. This is analogous to the way in which older video game systems can be emulated on (much faster) modern computers. 

Thus, if digital intelligence is invented that can think as fast as a human can, faster hardware would enable that same digital intelligence to think faster than a human. The speed of human thought would not be a stable resting place; it would be just one speed among many at which the digital intelligence could be run.

Copyability

Our colleague Steve Rayhawk likes to call digital intelligence “instant intelligence; just add hardware!” What Steve means is that while it will require extensive research to design the first digital intelligence, creating additional digital intelligences is just a matter of copying software. The population of digital minds can thus expand to fill the available hardware base, either through purchase (until the economic product of a new AI is less than the cost of the necessary computation) or through other means, for example hacking.

Depending on the hardware demands and the intelligence of initial digital intelligences, we might move fairly rapidly from the first human-level digital intelligence to a population of digital minds that outnumbers biological humans.

Copying also allows potentially rapid shifts in which digital intelligences, with which skills, fill the population. Since a digital intelligence’s skills are stored digitally, its exact current state can be copied, including memories and acquired skills. Thus if one digital intelligence becomes, say, 10% better at earning money per dollar of rentable hardware than other digital intelligences, then it could replace the others across the hardware base, for about a 10% gain in the economic productivity of those resources.

Digitality also opens up more parameters for variation. We can put humans through job-training programs, but we can’t do precise, replicable brain surgeries on them. Digital workers would probably be more editable than human workers are. In the case of whole brain emulation, we know that transcranial magnetic stimulation (TMS) applied to an area of the prefrontal cortex can improve working memory performance (Fregni et al. 2005). Since TMS works by temporarily decreasing or increasing the excitability of populations of neurons, it seems plausible that decreasing or increasing the “excitability” parameter of a certain population of (virtual) neurons in a digital mind could improve performance. We could also experimentally modify scores of other whole brain emulation parameters, e.g. simulated glucose levels, (virtual) fetal brain cells grafted onto particular brain modules, and rapid connections across different parts of the brain.¹ A modular, transparent AI could be even more directly editable than a whole brain emulation — for example via its source code.

Copyability thus dramatically changes the payoff from job training or other forms of innovation. If a human spends his summer at a job training program, his own future productivity is slightly boosted. But now, consider a “copy clan” — a set of copies of a single digital intelligence. If a copy clan of a million identical workers allocates one copy to such training, the learned skills can be copied to the rest of the copy clan, for a return on investment roughly a million times larger.

Goal coordination

Depending on the construction of the AIs, each instance within a copy clan could either: (a) have separate goals, heedless of the (indexically distinct) goals of its “copy siblings”; or (b) have a shared external goal that all instances of the AI care about, independently of what happens to their own particular copy. From the point of view of the AIs' creators, option (b) has obvious advantages in that the copy clan would not face the politics, principal agent problems, and other goal coordination problems that limit human effectiveness (Friedman 1993). A human who suddenly makes 500 times a subsistence income cannot use this to acquire 500 times as many productive hours per day. An AI of this sort, if its tasks are parallelizable, can.

Any gains made by such a copy clan, or by a human or human organization controlling that clan, could potentially be invested in further AI development, allowing initial advantages to compound. The ease of copying skills and goals across digital media thus seems to lead to a world in which agents' intelligence, productivity levels, and goals are unstable and prone to monoculture.

Further advantages to digital intelligence

How much room is there for design improvements to the human brain? Likely, quite a bit. As noted above, AI designers could seek improved hardware or more efficient algorithms to enable increased speed. They could also search for “qualitative” improvements — analogs of the difference between humans and chimpanzees or mice that enable humans to do tasks that are likely impossible for any number of chimpanzees or mice in any amount of time, such as engineering our way to the moon. AI designers can make use of several resources that were less accessible to evolution:

Increased serial depth. Due to neurons’ slow firing speed, the human brain relies on massive parallelization and is incapable of rapidly performing any computation that requires more than about 100 sequential operations (Feldman and Ballard 1982). Perhaps there are cognitive tasks that could be performed more efficiently and precisely if the organic brain’s ability to support parallelizable pattern-matching algorithms were supplemented by and integrated with support for fast sequential processes?

Increased real-time introspective access, high-bandwidth communication, or consciously editable algorithms. We humans access and revise our cognitive processes largely by fixed and limited pathways. Perhaps digitally recording states, consciously revising a larger portion of one’s thought patterns, or sharing high-bandwidth states with other agents would increase cognitive capacity?

Increased computational resources.  The human brain is small relative to the systems that could be built. The brain’s approximately 85–100 billion neurons is a size limit imposed not only by constraints on head size and metabolism, but also by the difficulty of maintaining integration between distant parts of the brain given the slow speed at which impulses can be transmitted along neurons (Fox 2011). While algorithms would need to be changed in order to be usefully scaled up, one can perhaps get a rough feel for the potential impact here by noting that humans have about [number] times the brain mass of chimps [citation], and that brain mass and cognitive ability correlate positively, with a correlation coefficient of about 0.4, in both humans and rats. [cite humans study, cite rats study]

Improved rationality. Some economists model humans as Homo economicus: self-interested rational agents who do what they believe will maximize the fulfillment of their goals (Friedman 1953). Behavioral studies, in contrast, suggest that we are more like Homer Simpson (Schneider 2010): we are irrational and non-maximizing beings that lack consistent, stable goals (Stanovich 2010; Cartwright 2011). But imagine if you were an instance of Homo economicus. You could stay on that diet, spend all your time learning which practices will make you wealthy, and then engage in precisely those, no matter how tedious or irritating they are. Some types of digital intelligence, especially transparent AI, could be written to be vastly more rational than humans, and accrue the benefits of rational thought and action. Indeed, the rational agent model (using Bayesian probability theory and expected utility theory) is a mature paradigm in current AI design (Russel and Norvig 2010, ch. 2).

Recursive “self”-improvement

Once a digital intelligence becomes better at AI design work than the team of programmers that brought it to that point, a positive feedback loop may ensue. Now when the digital intelligence improves itself, it improves the intelligence that does the improving. Thus, if mere human efforts suffice to produce digital intelligence this century, a large population of sped-up digital intelligences may be able to create a rapid cascade of self-improvement cycles, enabling a rapid transition. Chalmers (this volume) discusses this process in some detail, so here we will make only a few additional points.

The term “self,” in phrases like “recursive self-improvement” or “when the digital intelligence improves itself,” is something of a misnomer. The digital intelligence could conceivably edit its own code while it is running, but it could also create a new intelligence that runs independently. These “other” digital minds could perhaps be designed to have the same goals as the original, or to otherwise further its goals. In any case, the distinction between “self”-improvement and other AI improvement does not matter from the perspective of creating an intelligence explosion. The significant part is only that: (a) within a certain range, many digital intelligences probably can design intelligences smarter than themselves; and (b) given the possibility of shared goals, many digital intelligences will probably find greater intelligence useful (as discussed in more detail in section 4.1).

Depending on the abilities of the first digital intelligences, recursive self-improvement could either occur as soon as digital intelligence arrives, or it could occur after human design efforts, augmented by advantages from digitality, create a level of AI design competence that exceeds the summed research power of non-digital human AI designers. In any case, at least once self-improvement kicks in, AI development need not proceed on the timescale we are used to in human technological innovation. In fact, as discussed in the next section, the range of scenarios in which takeoff isn’t fairly rapid appears to be small, although non-negligible.

[the next section under 'from digital intelligence to explosion' is very large, and not included here]

¹ This third possibility is particularly interesting, in that many suspect that the slowness of cross-brain connections has been a major factor limiting the usefulness of large brains (Fox 2011).

Again, I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on. This section is less complete than the others; missing text is indicated with brackets: [].

_____

Many predictions of human-level digital intelligence have been wrong.¹ On the other hand, machines surpass human ability at new tasks with some regularity (Kurzweil 2005). For example, machines recently achieved superiority at visually identifying traffic signs at low resolution (Sermanet and LeCun 2011), diagnosing cardiovascular problems from some types of MRI scan images (Li et al. 2009), and playing Jeopardy! (Markoff 2011). Below, we consider several factors that, considered together, appear to increase the odds that we will develop digital intelligence as the century progresses.

More hardware. For at least four decades, computing power² has increased exponentially, in accordance with Moore’s law.³ Experts disagree on how much longer Moore’s law will hold (e.g. Mack 2011; Lundstrom 2003), but if it holds for two more decades then we may have enough computing power to emulate human brains by 2029.⁴ Even if Moore’s law fails to hold, our hardware should become much more powerful in the coming decades.⁵ More hardware doesn’t by itself give us digital intelligence, but it contributes to the development of digital intelligence in several ways:

Powerful hardware may improve performance simply by allowing existing “brute force” solutions to run faster (Moravec, 1976). Where such solutions do not yet exist, researchers might be incentivized to quickly develop them given abundant hardware to exploit. Cheap computing may enable much more extensive experimentation in algorithm design, tweaking parameters or using methods such as genetic algorithms. Indirectly, computing may enable the production and processing of enormous datasets to improve AI performance (Halevi et al., 2009), or result in an expansion of the information technology industry and the quantity of researchers in the field.⁶

Massive datasets. The greatest leaps forward in speech recognition and translation software have come not from faster hardware or smarter hand-coded algorithms, but from access to massive data sets of human-transcribed and human-translated words (Halevy, Norvig, and Pereira 2009). [add sentence about how datasets are expected to increase massively, or have been increasing massively and trends are expected to continue] [Possibly a sentence about Watson or usefulness of data for AI]

Better algorithms. Mathematical insights can reduce the computation time of a program by many orders of magnitude without additional hardware. For example, IBM’s Deep Blue played chess at the level of world champion Garry Kasparov in 1997 using about 1.5 trillion instructions per second (TIPS), but a program called Deep Junior did it in 2003 using only 0.15 TIPS. Thus, the power of the chess algorithms increased by a factor of 100 in only six years, or 3.33 orders of magnitude per decade (Richard and Shaw 2004). [add sentence about how this sort of improvement is not uncommon, with citations]

Progress in neuroscience. [neuroscientists have figured out brain algorithms X, Y, and Z that are related to intelligence.] New insights into how the brain achieves human-level intelligence can inform our attempts to build human-level intelligence with silicon (van der Velde 2010; Koene 2011). 

Accelerated science. A growing First World will mean that more researchers at well-funded universities will be available to do research relevant to digital intelligence. The world’s scientific output (in publications) grew by a third from 2002 to 2007 alone, much of this driven by the rapid growth of scientific output in developing nations like China and India (Smith 2011). New tools can accelerate particular fields, just as fMRI accelerated neuroscience in the 1990s. Finally, the effectiveness of scientists themselves can potentially be increased with cognitive enhancement drugs (Sandberg and Bostrom 2009) and brain-computer interfaces that allow direct neural access to large databases (Groß 2009). Better collaboration tools like blogs and Google scholar are already yielding results (Nielsen 2011).

Automated science. Early attempts at automated science — e.g., using data mining algorithms to make discoveries from existing data (Szalay and Gray 2006), or having a machine with no physics knowledge correctly infer natural laws from motion-tracking data (Schmidt and Lipson 2009) — were limited by the slowest part of the process: the human in the loop. Recently, the first “closed-loop” robot scientist successfully devised its own hypotheses (about yeast genomics), conducted experiments to test those hypotheses, assessed the results, and made novel scientific discoveries, all without human intervention (King et al. 2009). Current closed-loop robot scientists can only work on a narrow set of scientific problems, but future advances may allow for scalable, automated scientific discovery (Sparkes et al. 2010).

Embryo selection for better scientists. At age 8, Terrence Tao scored 760 on the math SAT, one of only [2?3?] children ever to do this at such an age; he later went on to [have a lot of impact on math]. Studies of similar kids convince researchers that there is a large “aptitude” component to mathematical achievement, even at the high end.⁷ How rapidly would mathematics or AI progress if we could create hundreds of thousands of Terrence Tao’s? This is a serious question because the creation of large numbers of exceptional scientists is an engineering project that we know in principle how to do. The plummeting costs of genetic sequencing [expected to go below AMOUNT per genome by SOONYEAR e.g. 2015] will soon make it feasible to compare the characteristics of an entire population of adults with those adults’ full genomes, and, thereby, to unravel the heritable components of intelligence, dilligence, and other contributors to scientific achievement. To make large numbers of babies with scientific abilities near the top of the current human range⁸ would then require only the ability to combine known alleles onto a single genome; procedures that can do this have already been developed for mice. China, at least, appears interested in this prospect.⁹

It isn’t clear which of these factors will ease progress toward digital intelligence, but it seems likely that — across a broad range of scenarios — some of these inputs will do so.

____

¹ For example, Simon (1965, 96) predicted that “machines will be capable, within twenty years, of doing any work a man can do.”

² The technical measure predicted by Moore’s law is the density of components on an integrated circuit, but this is closely tied to affordable computing power.

³ For important qualifications, see Nagy et al. (2010); Mack (2011).

⁴ This calculation depends on the “level of emulation” expected to be necessary for successful WBE. Sandberg and Bostrom (2008) report that attendees to a workshop on WBE tended to expect that emulation at the level of the brain’s spiking neural network, perhaps including membrane states and concentrations of metabolites and neurotransmitters, would be required for successful WBE. They estimate that if Moore’s law continues, we will have the computational capacity to emulate a human brain at the level of its spiking neural network by 2019, or at the level of metabolites and neurotransmitters by 2029.

⁵ Quantum computing may also emerge during this period. Early worries that quantum computing may not be feasible have been overcome, but it is hard to predict whether quantum computing will contribute significantly to the development of digital intelligence because progress in quantum computing depends heavily on unpredictable insights in quantum algorithms (Rieffel and Polak 2011).

⁶ Shulman and Sandberg (2010).

⁷ [Benbow etc. on study of exceptional talent; genetics of g; genetics of conscientiousness and openness, pref. w/ any data linking conscientiousness or openness to scientific achievement.  Try to frame in a way that highlights hard work type variables, so as to alienate people less.]

⁸ [folks with very top scientific achievement likely had lucky circumstances as well as initial gifts (so that, say, new kids with Einstein’s genome would be expected to average perhaps .8 times as exceptional).  However, one could probably identify genomes better than Einstein’s, both because these technologies would let genomes be combined that had unheard of, vastly statistically unlikely amounts of luck, and because e.g. there are likely genomes out there that are substantially better than Einstein (but on folks who had worse environmental luck).]

⁹ [find source]

_____

All references, including the ones used above:

• Bainbridge 2006 managing nano-bio-info-cogno innovations
• Baum Goertzel Goertzel 2011 how long until human-level ai
• Bostrom 2003 ethical issues in advanced artificial intelligence
• Legg 2008 machine super intelligence
• Caplan 2008 the totalitarian threat
• Sandberg & Bostrom 2011 machine intelligence survey
• Chalmers 2010 singularity philosophical analysis
• Turing 1950 machine intelligence
• Good 1965 speculations concerning...
• Von neumann 1966 theory of self-reproducing autonomata
• Solomonoff 1985 the time scale of artificial intelligence
• Vinge 1993 coming technological singularity
• Yudkowsky 2001 creating friendly ai
• Yudkowsky 2008a negative and positive factor in global risk
• Yudkowsky 2008b cognitive biases potentially affecting
• Russel Norvig 2010 artificial intelligence a modern approach 3e
• Nordman 2007 If and then: a critique of speculative nanoethics
• Moore and Healy the trouble with overconfidence
• Tversky Kahneman 2002 extensional versus intuitive reasoning, the conjunction fallacy
• Nickerson 1998 Confirmation Bias; A Ubiquitous Phenomenon in Many Guises
• Dreyfus 1972 what computers can't do
• Rhodes 1995 making of the atomic bomb
• Arrhenius 1896 On the Influence of Carbonic Acid in the Air Upon the Temperature
• Crawford 1997  Arrhenius' 1896 model of the greenhouse effect in context
• Rasmussen 1975 WASH-1400 report
• McGrayne 2011 theory that would not die
• Lundstrom 2003 Enhanced: Moore’s law forever? 
• Tversky and Kahneman 1974 Judgment under uncertainty: Heuristics and biases
• Horgan 1997 end of science
• Sutton and Barto 1998 reinforcement learning
• Hutter 2004 universal ai
• Schmidhuber 2007 godel machines
• Dewey 2011 learning what to value
• Simon 1965 The Shape of Automation for Men and Management
• Marcus 2008 kluge
• Sandberg Bostrom 2008 whole brain emulation
• Kurzweil 2005 singularity is near
• Sermanet Lecun 2011 traffic sign recognition with multi-scale convolutional networks
• Li et al. 2009 optimizing a medical image analysis system using
• Markoff 2011 watson trivial it's not
• Smith 2011 Knowledge networks and nations
• Sandberg Bostrom 2009 cognitive enhancement regulatory issues
• Groß 2009 Blessing or Curse? Neurocognitive Enhancement by “Brain Engineering”
• Williams 2011 prediction markets theory and appilcations
• Nielsen 2011 reinventing discovery
• Tetlock 2005 expert judgment
• Green & Armstrong 2007 The Ombudsman: Value of Expertise for Forecasting 
• Weinberg et al. 2010 philosophers expert intuiters
• Szalay and gray 2006 science in an exponential world
• Schmidt Lipson 2009 distilling free-form natural laws from experimental data
• King et al. 2009 the automation of science
• Sparkes et al. 2010 Towards Robot Scientists for autonomous scientific discovery
• Stanovich 2010 rationality and the reflective mind
• Lillienfeld, Ammirati, and Landfield 2009 giving debiasing away
• Lipman 1983 Thinking Skills Fostered by Philosophy for Children
• Fong et al 1986 The effects of statistical training on thinking about everyday problems
• Shoemaker (1979). The role of statistical knowledge in gambling decisions
• Larrick 2004 debiasing
• Gordon 2007 reasoning about the future of nanotechnology
• Landeta 2006 Current validity of the delphi method in social sciences
• Maddison 2001 the world economy a millenial perspective
• Niparko 2009 cochlear implants principles and practices
• Bostrom 2002 existential risks
• Joyce 2007 moral anti-realism stanford encyclopedia of philosophy
• Portmore 2011 commonsense consequentialism
• Martin 1971 brief proposal on immortality
• Bostrom Cirkovic 2008 global catastrophic risks
• National Academy of Sciences 2010 presistent forecasting of disruptive technologies
• Donohoe and Needham 2009 Moving best practice forward, Delphi characteristics
• Gordon 1994 the delphi method
• Kesten, Armstrong, and Graefe 2007 Methods to Elicit Forecasts from Groups
• Woudenberg 1991 an evaluation of delphi
• Armstrong 2006 Findings from evidence-based forecasting
• Armstrong 1985 Long-Range Forecasting: From Crystal Ball to Computer, 2nd edition
• Anderson and Anderson-Parente 2011 A case study of long-term Delphi accuracy
• Bixby 2002 Solving real-world linear programs: A decade and more of progress
• Fox 2011 the limits of intelligence
• Friedman 1953 The Methodology of Positive Economics
• Schneider 2010 homo economicus, or more like Homer Simpsons
• Cartwright 2011 behavioral economics
• Bacon and Van Dam 2010 recent progress in quantum algorithms
• Rieffel Polak 2011 quantum computing a gentle introduction
• Mack 2011 fifty years of moore’s law
• Nagy et al. 2010 testing laws of technological progress
• Lundstrom 2003 Moore’s law forever
• Shulman Sandberg 2010 implications of a software-limited singularity
• Moravec 1976 The Role of raw power in intelligence
• Halevi et al. 2009 The Unreasonable effectiveness of data
• Alberth 2008 forecasting technology costs via the experience curve
• Omohundro 2007 the nature of self-improving AI
• Kurzban 2011 why everyone (else) is a hypocrite: evolution and the modular mind
• Richard Shaw 2004 chips architectures and algorithms
• Yudkowsky 2010 timeless decision theory
• De Blanc Ontological Crises in Artificial Agents' Value Systems
• Dewey 2011 learning what to value
• Halevy, Norvig, and Pereira 2009 the unreasonable effectiveness of data
• Ramachandran 2011 the tell-tale brain
• van der Velde 2010 Where Artificial Intelligence and Neuroscience Meet
• Koene 2011 AGI and neuroscience: Open sourcing the brain (in AGI-11 proceedings)
• Lichtenstein, Fischoff, and Phillips 1982 calibration of probabilities the state of the art to 1980
• Griffin and Tversky 1992 The weighing of evidence and the determinants of confidence
• Yates, Lee, Sieck, Choi, Price 2002 Probability judgment across cultures
• Murphy and Winkler 1984 probability forecasting in meteorology
• Grove and Meehl 1996 Comparative Efficiency of Informal...
• Grove et al. 2000 Clinical versus mechanical prediction: A meta-analysis
• Kandel et al. 2000 principles of neural science, 4th edition
• Shulman 2010 Omohundro's "Basic AI Drives" and Catastrophic Risks
• Friedman 1993 Problems of Coordination in Economic Activity
• Cooke 1991 experts on uncertainty
• Yampolskiy forthcoming Leakproofing the Singularity
• Lampson 1973 a note on the confinement problem
• Schaeffer 1997 one jump ahead
• Dolan and Sharot 2011 Neuroscience of preference and choice

Again, I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on.

_____

From here to digital intelligence

Our first step is to survey the evidence suggesting that, barring global catastrophe and other disruptions to scientific progress,¹ there is a significant probability we will see the creation of digital intelligence² within a century.

Why focus on digital intelligence instead of, say, the cognitive enhancement of biological humans? As we discuss in a later section, digital intelligence has certain advantages (e.g. copyability) that make it likely to lead to intelligence explosion. 

Below, we discuss the different types of digital intelligence, what kinds of progress are likely to push us closer to digital intelligence, and how to estimate the time at which digital intelligence will arrive.

Types of digital intelligence

To count as a "digital intelligence," an artificial agent must have at least a human-level general capacity³ to achieve its goals in a wide range of environments, including novel ones.⁴

IBM's Jeopardy!-playing Watson computer is not a digital intelligence in this sense because it can only solve a narrow problem. Imagine instead a machine that can invent new technologies, manipulate humans with acquired social skills, and otherwise learn to navigate new environments on the fly. A digital intelligence need not be sentient, though, so long as it has a human-level capacity to achieve goals in a wide variety of environments.

There are many types of digital intelligence. To name just four:

• The code of a transparent AI is written explicitly by, and largely understood by, its programmers.⁵
• An opaque AI is not transparent to its creators. For example it could be, like the human brain, a messy ensemble of cognitive modules. In an AI, these modules might be written by different teams for different purposes using different languages and approaches.
• A whole brain emulation (WBE) is a computer emulation of the brain structures required to functionally reproduce human thought and perhaps consciousness (Sandberg and Bostrom 2008). We need not understand the detailed operation of a brain to reproduce it functionally on a computing substrate.
• A hybrid AI is a mix of any two or three of the above types of digital intelligence (transparent AI, opaque AI, and WBE).

Notes for this snippet

¹ By “disruptions to scientific progress” we have in mind “external” disruptions like catastrophe or a global totalitarianism that prevents the further progress of science (Caplan, 2008). We do not mean to include, for example, Horgan’s (1997) hypothesis that scientific progress may soon stop because there will be nothing left to discover that can be discovered, which we find unlikely.

² We introduce the term “digital intelligence” because we want a new term that refers to both human-level AI and whole brain emulations, and we don’t wish to expand the meaning of the common term "AI."

³ The notion of "human-level intelligence" is fuzzy, but nevertheless we can identify clear examples of intelligences below the human level (rhesus monkeys) and above the human level (a human brain running at 1000 times its normal speed). A human-level intelligence is any intelligent system not clearly below or above the human level.

⁴ Legg (2008) argues that many definitions of intelligence converge on this idea. We mean to endorse this informal definition, not Legg’s attempt to formalize intelligence in a later section of his manuscript.

⁵ Examples include many of today’s reinforcement learners (Sutton and Barto 1998), and also many abstract models such as AIXI (Hutter 2004), Gödel machines (Schmidhuber 2007), and Dewey’s (2011) “implemented agents.”

References for this snippet

• Sandberg & Bostrom 2008 whole brain emulation
• Caplan 2008 the totalitarian threat
• Horgan 1997 end of science
• Legg 2008 machine superintelligence
• Sutton & Barto 1998 reinforcement learning
• Hutter 2004 universal ai
• Schmidhuber 2007 godel machines
• Dewey 2011 learning what to value

I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on.

This snippet is a possible introduction to the analysis article. Its purpose is to show readers that we aim to take seriously some common concerns about singularity thinking, to bring readers into Near Mode about the topic, and to explain the purpose and scope of the article.

Note that the target style is serious but still more chatty than a normal journal article.

_____

The best answer to the question, "Will computers ever be as smart as humans?" is probably “Yes, but only briefly."

Vernor Vinge

Humans may create human-level artificial intelligence in this century (Bainbridge 2006; Baum, Goertzel, and Goertzel 2011; Bostrom 2003; Legg 2008; Sandberg and Bostrom 2011). Shortly thereafter, we may see an “intelligence explosion” or “technological Singularity” — a chain of events by which human-level AI leads, fairly rapidly, to intelligent systems whose capabilities far surpass those of biological humanity as a whole (Chalmers 2010).

How likely is this, and what should we do about it? Others have discussed these questions previously (Turing 1950; Good 1965; Von Neumann 1966; Solomonoff 1985; Vinge 1993; Yudkowsky 2001, 2008a; Russell and Norvig 2010, sec. 26.3); we will build on their thinking in our review of the subject.

Singularity Skepticism

Many are skeptical of Singularity arguments because they associate such arguments with detailed storytelling — the “if and then” fallacy of “speculative ethics” by which an improbable conditional becomes a supposed actual (Nordmann 2007). They are right to be skeptical: hundreds of studies show that humans are overconfident of their beliefs (Moore and Healy 2008), regularly overestimate the probability of detailed visualized scenarios (Tversky and Kahneman 2002), and tend to seek out only information that confirms their current views (Nickerson 1998). AI researchers are not immune from these errors, as evidenced by a history of over-optimistic predictions going back to the 1956 Dartmouth conference on AI (Dreyfus 1972).

Nevertheless, mere mortals have at times managed to reason usefully and somewhat accurately about the future, even with little data. When Leo Szilard conceived of the nuclear chain reaction, he realized its destructive potential and filed his patent in a way that kept it secret from the Nazis (Rhodes 1995, 224–225). Svante Arrhenius' (1896) models of climate change lacked modern climate theory and data but, by making reasonable extrapolations from what was known of physics, still managed to predict (within 2°C) how much warming would result from a doubling of CO₂ in the atmosphere (Crawford 1997). Norman Rasmussen's (1975) analysis of the safety of nuclear power plants, written before any nuclear accidents had occurred, correctly predicted several details of the Three Mile Island incident that previous experts had not (McGrayne 2011, 180).

In planning for the future, how can we be more like Rasmussen and less like the Dartmouth conference? For a start, we can apply the recommendations of cognitive science on how to meliorate overconfidence and other biases (Larrick 2004; Lillienfeld, Ammirati, and Landfield 2009). In keeping with these recommendations, we acknowledge unknowns and do not build models that depend on detailed storytelling. For example, we will not assume the continuation of Moore’s law, nor that hardware trajectories determine software progress. To avoid nonsense, it should not be necessary to have superhuman reasoning powers; all that should be necessary is to avoid believing we know something when we do not.

One might think such caution would prevent us from concluding anything of interest, but in fact it seems that intelligence explosion may be a convergent outcome of many or most future scenarios. That is, an intelligence explosion may have fair probability, not because it occurs in one particular detailed scenario, but because, like the evolution of eyes or the emergence of markets, it can come about through many different paths and can gather momentum once it gets started. Humans tend to underestimate the likelihood of such “disjunctive” events, because they can result from many different paths (Tversky and Kahneman 1974). We suspect the considerations in this paper may convince you, as they did us, that this particular disjunctive event (intelligence explosion) is worthy of consideration.

First, we provide evidence which suggests that, barring global catastrophe and other disruptions to scientific progress, there is a significant probability we will see the creation of digital intelligence within a century. Second, we suggest that the arrival of digital intelligence is likely to lead rather quickly to intelligence explosion. Finally, we discuss the possible consequences of an intelligence explosion and which actions we can take now to influence those results.

These questions are complicated, the future is uncertain, and our chapter is brief. Our aim, then, can only be to provide a quick survey of the issues involved. We believe these matters are important, and our discussion of them must be permitted to begin at a low level because there is no other place to lay the first stones.

References for this snippet

• Bainbridge 2006 managing nano-bio-info-cogno innovations
• Baum Goertzel Goertzel 2011 how long until human-level ai
• Bostrom 2003 ethical issues in advanced artificial intelligence
• Chalmers 2010 singularity philosophical analysis
• Legg 2008 machine super intelligence
• Sandberg & Bostrom 2011 machine intelligence survey
• Turing 1950 machine intelligence
• Good 1965 speculations concerning...
• Von neumann 1966 theory of self-reproducing autonomata
• Solomonoff 1985 the time scale of artificial intelligence
• Vinge 1993 coming technological singularity
• Yudkowsky 2001 creating friendly ai
• Yudkowsky 2008a negative and positive factor in global risk
• Russel Norvig 2010 artificial intelligence a modern approach 3e
• Nordman 2007 If and then: a critique of speculative nanoethics
• Moore and Healy the trouble with overconfidence
• Tversky Kahneman 2002 extensional versus intuitive reasoning, the conjunction fallacy
• Nickerson 1998 Confirmation Bias; A Ubiquitous Phenomenon in Many Guises
• Dreyfus 1972 what computers can't do
• Rhodes 1995 making of the atomic bomb
• Arrhenius 1896 On the Influence of Carbonic Acid in the Air Upon the Temperature
• Crawford 1997  Arrhenius' 1896 model of the greenhouse effect in context
• Rasmussen 1975 WASH-1400 report
• McGrayne 2011 theory that would not die
• Larrick 2004 debiasing
• Lillienfeld, Ammirati, and Landfield 2009 giving debiasing away
• Tversky and Kahneman 1974 Judgment under uncertainty: Heuristics and biases

A few months ago, Anna Salamon and I began to write an academic overview of intelligence explosion scenarios — something we could hand to people to explain all our major points in one brief article.

We encountered two major problems.

First: The Summit happened, taking all of our time. Then I was made Executive Director, taking all of my time in a more persistent way.

Second: Being thorough and rigorous in an overview of intelligence explosion requires deep knowledge of a huge spectrum of science and philosophy: history of AI progress, history of planning for the future mattering, AI architectures, hardware progress, algorithms progress, massive datasets, neuroscience, factors in the speed of scientific progress, embryo selection, whole brain emulation, properties of digital minds, AI convergent instrumental values, self-improvement dynamics, takeoff scenarios, heuristics and biases, unipolar and multipolar intelligence explosion scenarios, human values and value extrapolation, decision theory, arms races, human dynamics of technological development, technological forecasting, the economics of machine intelligence, anthropics, evolution, AI-boxing, and much more. Because we were trying to write a short article, we kept having to consume and compress an entire field of knowledge into a single paragraph (or even a single sentence!) with the perfect 2-8 citations, which occasionally meant several days of work for a single paragraph. (This is an extreme example, but it's the kind of problem we often encountered, in different degrees.)

So, we've decided to take a different approach and involve the broader community.

We'll be posting short snippets, short pieces of the puzzle, for feedback from the community. Sometimes we'll pose questions, or ask for references about a given topic, or ask for suggested additions to the dialectic we present.

In the end, we hope to collect and remix the best and most essential snippets, incorporate the feedback and additions provided by the community, and write up the final article.

Think of it as a Polymath Project for intelligence explosion analysis. It's collaborative science and philosophy. Members of Less Wrong tend to be smart, and each one has deep knowledge of one or a few fields that we may not have. We hope you'll join us, and contribute your expertise to this project.

I'll keep a table of contents of all the snippets here, as they are published.

Draft #1:

1. Introduction
2. Types of digital intelligence
3. Why designing digital intelligence gets easier over time
4. How long before digital intelligence?
5. From digital intelligence to intelligence explosion
6. [not finished]

Also see:

• Is intelligence explosion a disjunctive or conjunctive event?