# Q&A with Michael Littman on risks from AI

15 19 December 2011 09:51AM

Michael L. Littman is a computer scientist. He works mainly in reinforcement learning, but has done work in machine learning, game theory, computer networking, Partially observable Markov decision process solving, computer solving of analogy problems and other areas. He is currently a professor of computer science and department chair at Rutgers University.

Homepage: cs.rutgers.edu/~mlittman/

### The Interview:

Michael Littman: A little background on me.  I've been an academic in AI for not-quite 25 years.  I work mainly on reinforcement learning, which I think is a key technology for human-level AI---understanding the algorithms behind motivated behavior.  I've also worked a bit on topics in statistical natural language processing (like the first human-level crossword solving program).  I carried out a similar sort of survey when I taught AI at Princeton in 2001 and got some interesting answers from my colleagues.  I think the survey says more about the mental state of researchers than it does about the reality of the predictions.

In my case, my answers are colored by the fact that my group sometimes uses robots to demonstrate the learning algorithms we develop.  We do that because we find that non-technical people find it easier to understand and appreciate the idea of a learning robot than pages of equations and graphs.  But, after every demo, we get the same question: "Is this the first step toward Skynet?"  It's a "have you stopped beating your wife" type of question, and I find that it stops all useful and interesting discussion about the research.

Anyhow, here goes:

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Michael Littman:

10%: 2050 (I also think P=NP in that year.)
50%: 2062
90%: 2112

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Michael Littman: epsilon, assuming you mean: P(human extinction caused by badly done AI | badly done AI)

I think complete human extinction is unlikely, but, if society as we know it collapses, it'll be because people are being stupid (not because machines are being smart).

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Michael Littman: epsilon (essentially zero).  I'm not sure exactly what constitutes intelligence, but I don't think it's something that can be turbocharged by introspection, even superhuman introspection.  It involves experimenting with the world and seeing what works and what doesn't.  The world, as they say, is its best model.  Anything short of the real world is an approximation that is excellent for proposing possible solutions but not sufficient to evaluate them.

Q3-sub: P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?

Michael Littman: Ditto.

Q3-sub: P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?

Michael Littman: 1%. At least 5 years is enough for some experimentation.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Michael Littman: No, I don't think it's possible.  I mean, seriously, humans aren't even provably friendly to us and we have thousands of years of practice negotiating with them.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Michael Littman: In terms of science risks (outside of human fundamentalism which is the only non-negligible risk I am aware of), I'm most afraid of high energy physics experiments, then biological agents, then, much lower, information technology related work like AI.

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

Michael Littman: I think people are currently hypersensitive.  As I said, every time I do a demo of any AI ideas, no matter how innocuous, I am asked whether it is the first step toward Skynet.  It's ridiculous.  Given the current state of AI, these questions come from a simple lack of knowledge about what the systems are doing and what they are capable of.  What society lacks is not a lack of awareness of risks but a lack of technical understanding to *evaluate* risks.  It shouldn't just be the scientists assuring people everything is ok.  People should have enough background to ask intelligent questions about the dangers and promise of new ideas.

Q7: Can you think of any milestone such that if it were ever reached you would expect human窶人evel machine intelligence to be developed within five years thereafter?

Michael Littman: Slightly subhuman intelligence?  What we think of as human intelligence is layer upon layer of interacting subsystems.  Most of these subsystems are complex and hard to get right.  If we get them right, they will show very little improvement in the overall system, but will take us a step closer.  The last 5 years before human intelligence is demonstrated by a machine will be pretty boring, akin to the 5 years between the ages of 12 to 17 in a human's development.  Yes, there are milestones, but they will seem minor compared to first few years of rapid improvement.

Sort By: Old
Comment author: 19 December 2011 10:28:33AM 3 points [-]

The last 5 years before human intelligence is demonstrated by a machine will be pretty boring, akin to the 5 years between the ages of 12 to 17 in a human's development.

Those were some of the most exciting years of my life.

Similarly, I expect the run up to machine intelligence to consist of interesting times.

Comment author: 19 December 2011 10:34:48AM *  0 points [-]

I'm not sure exactly what constitutes intelligence, but I don't think it's something that can be turbocharged by introspection, even superhuman introspection. It involves experimenting with the world and seeing what works and what doesn't.

A common sentiment. Shane Legg even says something similar:

We then use this fact to prove that although very powerful prediction algorithms exist, they cannot be mathematically discovered due to Godel incompleteness. Given how fundamental prediction is to intelligence, this result implies that beyond a moderate level of complexity the development of powerful artificial intelligence algorithms can only be an experimental science.

I think we can now label the need for an environment as a fallacy. Most of the guts of building an intelligent agent involves finding good computable approximations to Solomonoff induction - and you can do that pretty well in a virtual world with an optimisation algorithm and a fitness function based around something like AIQ. This is essentiallly a math problem.

Comment author: 20 December 2011 12:53:48PM 4 points [-]

I think we can now label the need for an environment as a fallacy.

I think it's difficult to label something as a fallacy when there is almost no hard evidence either way about who is right. The vast majority of AI researchers (including myself) don't think that Solomonoff induction will solve AI. It is also possible to construct formal environments where performing better than chance is impossible without constant interaction with the environment (unless P=NP). So, if the problem is essentially a math problem, that would have to depend on specific facts about the world that make it different from such environments.

Comment author: 20 December 2011 02:30:05PM 0 points [-]

The vast majority of AI researchers (including myself) don't think that Solomonoff induction will solve AI.

Solomonoff induction certainly doesn't give you an evaluation function on a plate. It would pass the Turing test, though. If the "vast majority of AI researchers" don't realise that, they should look into the issue further.

It is also possible to construct formal environments where performing better than chance is impossible without constant interaction with the environment (unless P=NP). So, if the problem is essentially a math problem, that would have to depend on specific facts about the world that make it different from such environments.

So: Occam's razor - the foundation of science - is also needed. We know about that. Other facts about the world might help a little (arguably, some elements relating to locality are implicit in the reference machine) - but they don't seem to be as critical.

Comment author: 20 December 2011 03:07:34PM 1 point [-]

So: Occam's razor - the foundation of science - is also needed.

I was referring to computational issues, not whether a complexity prior is reasonable or not. It is possible that making inferences about the environment requires you to solve hard computational problems, and that these problems become easier after additional interaction with the environment. I don't see how Occam's razor suggests that our world doesn't look like that (in fact, I currently think that our world does look like that, although my confidence in this is not very high).

Comment author: 20 December 2011 03:51:57PM 0 points [-]

It is possible that making inferences about the environment requires you to solve hard computational problems, and that these problems become easier after additional interaction with the environment.

Well, of course - but that's learning - which Solomonoff induction models just fine (it is a learning theory).

Or maybe you are suggesting that organisms modify their environment to make their problems simpler. That is perfectly possible - but I don't really see how it is relevant.

You apparently didn't disagree with Solomonoff induction allowing the Turing test to be passed. So: what exactly is your beef with its centrality and significance?

Comment author: 20 December 2011 04:05:35PM 1 point [-]

It's possible I misunderstood your original comment. Let me play it back to you in my own words to make sure we're on the same page.

My understanding was that you did not think it would be necessary for an AGI to interact with its environment in order to achieve superhuman intelligence (or perhaps that a limited initial interaction with its environment would be sufficient, after which it could just go off and think). Is that correct, or not?

P.S. I think that I also disagree with the Solomonoff induction -> Turing test proposition; but I'd rather delay discussing that point because I think it is contingent on the others.

Comment author: 20 December 2011 07:25:18PM *  1 point [-]

My understanding was that you did not think it would be necessary for an AGI to interact with its environment in order to achieve superhuman intelligence (or perhaps that a limited initial interaction with its environment would be sufficient, after which it could just go off and think). Is that correct, or not?

Pretty much. Virtual environments are fine, contain lots of complexity (chaos theory) and have easy access to lots of interesting and difficult problems (game of go, etc). Virtual worlds permit the development of intelligent agents just like the "real" world does. A good job too - since we have no terribly good way of telling whether our world exists under simulation or not.

The Solomonoff induction -> Turing test proposition is detailed here.

Comment author: 28 December 2011 05:04:55PM 0 points [-]

Sorry for the delayed response, it took me a while to get through the article and corresponding Hutter paper. Do you know of any sources that present the argument for why the Kolmogorov complexity of the universe should be relatively low (i.e. not proportional to the number of atoms), or else why Solomonoff induction would perform well even if the Kolmogorov complexity is high? These both seem intuitively true to me, but I feel uneasy accepting them as fact without a solid argument.

Comment author: 29 December 2011 12:02:30AM *  0 points [-]

The Kolmogorov complexity of the universe is a totally unknown quantity - AFAIK. Yudkowsky suggests a figure of 500 bits here - but there's not much in the way of supporting argument.

Solomonoff induction doesn't depend of the Kolmogorov complexity of the universe being low. The idea that Solomonoff induction has something to do with the Kolmogorov complexity of the universe seems very strange to me.

Instead, consider that Solomonoff induction is a formalisation of Occam's razor - which is a well-established empirical principle.

Comment author: 29 December 2011 12:24:18AM 2 points [-]

I don't understand. I thought the point of Solomonoff induction is that its within an additive constant of being optimal, where the constant depends on the Kolmogorov complexity of the sequence being predicted.

Comment author: 29 December 2011 12:24:33AM *  0 points [-]

The idea that Solomonoff induction has something to do with the Kolmogorov complexity of the universe seems very strange to me.

Wouldn't it put an upper bound on the complexity of any given piece, as you can describe it with "the universe, plus the location of what I care about"?

Edited to add: Ah, yes but "the location of what I care about" is has potentially a huge amount of complexity to it.

Comment author: 19 December 2011 10:35:47AM *  6 points [-]

2050 (I also think P=NP in that year.)

P=NP - WTF?!? ;-)

Comment author: 19 December 2011 11:07:35AM 1 point [-]

P=NP - WTF?!? ;-)

Also see: Polls And Predictions And P=NP

Comment author: 19 December 2011 01:23:52PM *  3 points [-]

Does anyone know what the largest amount of money wagered on this question is?

EDIT: I'm aware of a few bets on specific claimed proofs, but have not been able to find any bets on the general question that exceed a few hundred dollars (unless you count the million-dollar prizes various institutes are offering).

Comment author: 19 December 2011 03:02:50PM 4 points [-]

Does anyone know what the largest amount of money wagered on this question is?

Don't know, but Scott Aaronson once bet \$200,000 on a proof being wrong. He wrote:

I’ve literally bet my house on it.

Comment author: 19 December 2011 04:04:05PM 2 points [-]

When I read that, I didn't expect him to actually pay up in the unlikely event the proof was right - there's a big difference between saying 'I bet my house' on your blog and actually sending a few hundred thousand or million bucks to the Long Now Foundation's Long Bets project.

Comment author: 04 January 2012 09:09:01AM *  0 points [-]

When I read that, I didn't expect him to actually pay up in the unlikely event the proof was right

Likewise.

sending a few hundred thousand or million bucks to the Long Now Foundation's Long Bets project.

With Long Bets you lose the money (to your chosen charity) even if you are right, so not an ideal comparison.

Comment author: 04 January 2012 02:48:36PM *  0 points [-]

It was an example of a more credible commitment than a blog post. To paraphrase Buffet's Noah principle, 'predicting rain doesn't count, building arks does'.

EDIT: an additional disadvantage to Long Bets is that they stash the stakes in a very low return fund (but one that should be next to invulnerable). Depending on your views about the future and your investment abilities, the opportunity cost could be substantial.

Comment author: 19 December 2011 02:33:04PM *  2 points [-]

By far that comment is the one that is farthest outside his expertise. I'm not sure why he's commenting on it. (He is a computer scientist but none of his work seems to be in complexity theory or is even connected to it as far as I can tell.) But he's still very respected and I would presume knows a lot about issues in parts of compsci that aren't his own area of research. It is possible that he made a typo?

Comment author: 15 January 2012 03:10:40PM 6 points [-]

Not a typo---I was mostly being cheeky. But, I have studied complexity theory quite a bit (mostly in analyzing the difficulty of problems in AI) and my 2050 number came from the following thought experiment. The problem 3-SAT is NP complete. It can be solved in time 2^n (where n is the number of variables in the formula). Over the last 20 or 30 years, people have created algorithms that solve the problem in c^n for ever decreasing values of c. If you plot the rate of decrease of c over time, it's a pretty good fit (or was 15 years ago when I did this analysis) for a line that goes below 1 in 2050. (If that happens, an NP-hard problem would be solvable in polynomial time and thus P=NP.) I don't put much stake in the idea that the future can be predicted by a graph like that, but I thought it was fun to think about. Anyhow, sorry for being flip.

Comment author: 15 January 2012 05:41:53PM 1 point [-]

Thanks for clarifying. (And welcome to Less Wrong.)

Comment author: 21 January 2012 03:36:55PM 2 points [-]

Side note: I did this analysis initially in honor of Donald Loveland (a colleague at the time whose satisfiability solver sits at the root of this tree of discoveries). I am gratified to see that he was interviewed on lesswrong on a more recent thread!

Comment author: 19 December 2011 12:43:00PM *  20 points [-]

I think this expert is anthropomorphizing too much. To pose an extinction risk, a machine doesn't even need to talk, much less replicate all the accidental complexity of human minds. It just has to be good at physics and engineering.

These tasks seem easier to formalize than many other things humans do: in particular, you could probably figure out the physics of our universe from very little observational data, given a simplicity prior and lots of computing power (or a good enough algorithm). Some engineering tasks are limited by computing power too, e.g. protein folding is an already formalized problem, and a machine that could solve it efficiently could develop nanotech faster than humans do.

We humans probably suck at physics and engineering on an absolute scale just like we suck at multiplying 32-bit numbers, see Moravec's paradox. And we probably suck at these tasks about as much as it's possible to suck and still build a technological civilization, because otherwise we would have built it at an earlier point in our evolution.

We now know that playing chess doesn't require human-level intelligence as Littman understands it. It may turn out that destroying the world doesn't require human-level intelligence either. A narrow AI could do just fine.

Comment author: 19 December 2011 01:00:58PM 4 points [-]

I wouldn't take Moravec's paradox too seriously; all it seems to indicate is that we're better at programming a system we've spent thousands of years formalizing (eg, math) than a system that's built into our brains so that we never really think about it...hardly surprising to me.

Comment author: 19 December 2011 01:13:35PM *  7 points [-]

I think Moravec's paradox is more than a selection effect. Face recognition requires more computing power than multiplying two 32-bit numbers, and it's not just because we've learned to formalize one but not the other. We will never get so good at programming computers that our face-recognition programs get faster than our number-multiplication programs.

Comment author: 19 December 2011 01:23:26PM 8 points [-]

We now know that playing chess doesn't require human-level intelligence as Littman understands it. It may turn out that destroying the world doesn't require human-level intelligence either. A narrow AI could do just fine.

Interesting: this framing moved me more than your previous explanation.

Comment author: 19 December 2011 01:57:19PM 3 points [-]

And we probably suck at these tasks about as much as it's possible to suck and still build a technological civilization, because otherwise we would have built it at an earlier point in our evolution.

<Expression of extravagant agreement and emphasis/>

Comment author: 19 December 2011 02:13:08PM 1 point [-]

This is a well-known argument. I got it from Eliezer somewhere, don't remember where.

Comment author: 19 December 2011 02:38:02PM *  7 points [-]

This is a well-known argument. I got it from Eliezer somewhere, don't remember where.

Yes, and I'm sick of trying to explain to people why "we have no evidence that it is possible to have higher than human intelligence" is trivially absurd for approximately this reason. Hence encouragement of others saying the same thing.

Comment author: 19 December 2011 04:00:00PM 1 point [-]

I'm sick of trying to explain to people why "we have no evidence that it is possible to have higher than human intelligence" is trivially absurd for approximately this reason.

You wrote a reference post where you explain why you would deem anyone who wants to play quantum roulette crazy. If the argument mentioned by cousin_it is that good and you have to explain it to people that often, I want you to consider writing a post on it where you outline the argument in full detail ;-)

You could start by showing how most evolutionary designs are far short of their maximum efficiency and that we therefore have every reason to believe that human intelligence barely reached the minimum threshold necessary to build a technological civilization.

Comment author: 19 December 2011 04:03:11PM *  1 point [-]

You could start by showing how most evolutionary designs are far short of their maximum efficiency and that we therefore have every reason to believe that human intelligence barely reached the minimum threshold necessary to build a technological civilization.

The therefore is pointing the wrong direction. That's the point!

Comment author: 19 December 2011 04:29:48PM *  0 points [-]

The therefore is pointing the wrong direction. That's the point!

Human intelligence barely reached the minimum threshold necessary to build a technological civilization and therefore we have every reason to believe that most evolutionary designs are far short of their maximum efficiency? That seems like a pretty bold claim based on the the fact that some of our expert systems are better at narrow tasks that were never optimized for by natural selection.

If you really want to convince people that human intelligence is the minimum of general intelligence possible given the laws of physics then in my opinion you have to provide some examples of other evolutionary designs that are very inefficient compared to their technological counterparts.

Comment author: 19 December 2011 04:39:06PM -2 points [-]

Cousin_it, this is why I am glad to see people other than myself explaining the concept. I just don't have the patience to deal with this kind of thinking.

Comment author: 19 December 2011 05:18:48PM *  3 points [-]

I don't see how us not having build a technological civilization earlier in our history does constitute evidence that we only have the minimum intelligence that is necessary to do so. I don't think that intelligence makes as much of a difference to how quickly discoveries are made as you seem to think.

...this is why I am glad to see people other than myself explaining the concept.

I have never seen you explain the concept nor have I seen you refer to an explanation. I must have missed that, but I also haven't read all of your comments.

Comment author: 20 December 2011 03:13:35PM 3 points [-]

I don't understand. XiXiDu's thinking was "if your assertion about humans was true, then we would expect to see these other things as well (i.e., other species being minimally fit for a task when they first start doing it); we therefore have a way of testing this hypothesis in a fairly convincing way, why don't we actually do that so that we can see if we're right or not?" That seems to me like the cornerstone of critical thought, or am I missing what you found objectionable?

Comment author: 20 December 2011 04:46:54PM *  0 points [-]

..."if your assertion about humans was true, then we would expect to see these other things as well (i.e., other species being minimally fit for a task when they first start doing it); we therefore have a way of testing this hypothesis in a fairly convincing way,..."

That is a good suggestion and I endorse it. I have however been thinking about something else.

I suspected that people like cousin_it and wedrifid must base their assumption that human intelligence is close to the minimum level of efficiency (optimization power/resources used) on other evidence, e.g. that expert systems can factor numbers 10^180 times faster than humans can. I didn't think that the whole argument rests on the fact that humans didn't start to build a technological civilization right after they became mentally equipped to do so.

Don't get a wrong impression here, I agree that it is very unlikely that human intelligence is close to the optimum. But I also don't see that we have much reason to believe that it is close to the minimum. Further I believe that intelligence is largely overrated by some people on lesswrong and that conceptual revolutions, e.g. the place-value notation method of encoding numbers, wouldn't have been discovered much quicker by much more intelligent beings other than due to lucky circumstances. In other words, I think that the speed of discovery is not proportional to intelligence but rather that intelligence quickly hits diminishing returns (anecdotal evidence here includes that real world success doesn't really seem to scale with IQ points. Are people like Steve Jobs that smart? Could Terence Tao become the richest person if he wanted to? Are high karma people on lesswrong unusually successful?).

But I digress. My suggestion was to compare technological designs with evolutionary designs. For example animal echolocation with modern sonar, ant colony optimization algorithm with the actual success rate of ant behavior, energy efficiency and maneuverability of artificial flight with insect or bird flight...

If intelligence is a vastly superior optimization process compared to evolution then I suspect that any technological replications of evolutionary designs, that have been around for some time, should have been optimized to an extent that their efficiency vastly outperforms that of their natural counterparts. And from this we could then draw the conclusion that intelligence itself is unlikely to be an outlier but just like those other evolutionary designs very inefficient and subject to strong artificial amplification.

ETA: I believe that even sub-human narrow AI is an existential risk. So that I believe that lots of people here are hugely overconfident about a possible intelligence explosion doesn't really change that much with respect to risks from AI.

Comment author: 19 December 2011 10:45:30PM *  4 points [-]

If you really want to convince people that human intelligence is the minimum of general intelligence possible given the laws of physics then in my opinion you have to provide some examples of other evolutionary designs that are very inefficient compared to their technological counterparts.

E. g. trying to estimate how fast the first animal that walked on land could run and comparing that with how fast the currently fastest animal on land can run, how fast the fastest robot with legs can run and how fast the fastest car can "run"?

Comment author: 19 December 2011 04:25:11PM *  7 points [-]

And we probably suck at these tasks about as much as it's possible to suck and still build a technological civilization, because otherwise we would have built it at an earlier point in our evolution.

I don't think this follows. Humans spent thousands of years in near stagnation (the time before the dawn of agriculture is but one example). It isn't clear what caused technological civilization to take off but when new discoveries occurred almost looks like some sort of nearly random process except that the probability of a new discovery or invention increases as more discoveries occur. I'd almost consider modeling it as a biased coin which starts off with an extreme bias towards tails, but each time it turns up heads, the bias shifts a bit in the heads direction. Something like P(heads on the nth flip)= (1+k)/(10^5 + k) where k is the number of previous flips that came up heads.If that's the case, then the timing doesn't by itself tell us much about where our capacity is for civilization. It doesn't look that improbable that some other extinct species might even have had the capability at about where we are or higher but went extinct before they got those first few lucky coin flips.

Comment author: 19 December 2011 05:44:08PM *  0 points [-]

In a blink of evolution's eye.

Comment author: 19 December 2011 06:31:20PM 3 points [-]

almost looks like some sort of nearly random process except that the probability of a new discovery or invention increases as more discoveries occur.

And as population increases that would tend to increase the rate of discovery or invention as well. This is basically Julian Simon's argument in The Great Breakthrough and Its Causes, that gradually increasing population hit a point where the rates of discovery and invention suddenly started increasing rapidly (and population then started increasing even more rapidly), resulting in the Renaissance and ultimately in the Industrial Revolution. He gives some thought and argument as to why they didn't happen earlier in India or China, but I think the specific arguments a bit iffy.

Comment author: 19 December 2011 09:53:58PM 6 points [-]

It just has to be good at physics and engineering.

I would contend it would have to know what is in the current environment as well. What bacteria and other micro organisms it is likely to face ( a largely unexplored question by humans), what chemicals it will have available (as potential feedstocks and poisons) and what radiation levels.

To get these from first principles it would have to recreate the evolution of earth from scratch.

Some engineering tasks are limited by computing power too, e.g. protein folding is an already formalized problem,

What do you mean by a formalized problem in this context? I'm interested in links on the subject.

Comment author: 19 December 2011 10:06:59PM *  3 points [-]

There are a variety of formalized versions of protein folding. See for example this paper(pdf). There are however questions if these models are completely accurate. Computing how a protein will fold based on a model is often so difficult that testing the actual limits of the models is tricky. The model given in the paper I linked to is known to be too simplistic in many practical cases.

Comment author: 19 December 2011 11:12:23PM *  2 points [-]

Sorry for speaking so confidently. I don't really know much about protein folding, it was just the impression I got from Wikipedia: 1, 2.

Comment author: 19 December 2011 01:04:58PM *  4 points [-]

No, I don't think it's possible. I mean, seriously, humans aren't even provably friendly to us and we have thousands of years of practice negotiating with them.

Not sure this is a fair comparison for 2 reasons: 1) We don't have the complete source code to human consciousness yet, so we can't do a good analysis of it, and 2) If anything primates are provably unfriendly to each other (at least outside their tribal group).

EDIT: Yes, I realize that a human genome is sort of a source code to our behavior, but having it without a complete theory of physics is rather like being given the source code to an AI in an unknown format.

Comment author: 19 December 2011 05:20:17PM *  4 points [-]

Yes, I realize that a human genome is sort of a source code to our behavior, but having it without a complete theory of physics is rather like being given the source code to an AI in an unknown forma

Having the exact laws of physics here probably doesn't matter as much as simply having a better understanding of human development. The genome isn't all that matters. What proteins are in the egg at the start matter a lot, and there are things like epigenetics. And the computational level involved in trying to model anything in the human body reliably is immense. The fundamental laws of physics probably don't matter much for human behavior.

Comment author: 19 December 2011 03:27:53PM 7 points [-]

Q3-sub: P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?

Michael Littman: 1%. At least 5 years is enough for some experimentation.

That's the answer that surprised me the most. I'm willing to defer to his experience when it comes to the feasibility of human-level AI itself, but human-level AI + a blueprint of how it was built + better resources than a human in terms of raw computing power and memory + having a much closer interface to code than a human does + self-modification .... well, all that seems like a pretty straightforward recipe for creating superhuman intelligence.

Comment author: 19 December 2011 10:37:39PM 4 points [-]

Lots of Machine Learning programs have parameters set to certain values because they seem to work well (e.g. update rates on peceptrons). Perhaps he is extrapolating that into full AI. So the blueprint would be strewn with comments like "Set complexity threshold to attributing external changes to volitional agents to 0.782. Any higher and the agent believes humans aren't intelligent and tries (and fails) to predict them from first principles rather than the intentional stance. Any lower and the agent believes rocks are intelligent and just want to stay still. Also this interferes with learning rate alpha for unknown reasons".

So experimentation with different variants of values might take significant time to evaluate their efficacy (especially if you have to raise from a baby each time).

I'm also guessing that Michael doesn't think that AI's are likely to be malicious and write malware to run experiments in the darknet :)

Comment author: 20 December 2011 02:39:46PM *  3 points [-]

This is the reason why I'm more worried about hardware overhang than recursive self-improvement. Currently known learning algorithms seem to all have various parameters like that whose right value you can't know a priori - you have to experiment to find out. And when setting parameter 420 to .53 gives you a different result than setting it to .48, you don't necessarily know which result is more correct, either. You need some external way of verifying the results, and you need to be careful that you are still interpreting the external data correctly and didn't just self-modify yourself to go insane. (You can test yourself on data you've generated yourself, and where you know the correct answers, but that doesn't yet show that you'll process real-world data correctly.)

My current intuition suggests that general intelligence is horribly fragile, in the sense that it's an extremely narrow slice of mindspace that produces designs that actually reason correctly. Just like with humans, if you begin to tamper with your own mind, you're most likely to do damage if you don't know what you're doing - and evolution has had time to make our minds quite robust in comparison.

That isn't to say that an AGI couldn't RSI itself to godhood in a relatively quick time, especially if it had humans scientists helping it out. Also, like cousin_it pointed out, you don't necessarily need superintelligence to destroy humanity. But the five year estimate doesn't strike me as unreasonable.

What I suspect - and hope, since it might give humanity a chance - to happen is that some AGI will begin a world-takeover attempt, but then fail due to some epistemic equivalent of a divide-by-zero error, falling prey to Pascal's mugging or something.

Then again, it might fail, but only after having destroyed humans while in the process.

Comment author: 20 December 2011 04:52:43PM 1 point [-]

You need some external way of verifying the results, and you need to be careful that you are still interpreting the external data correctly and didn't just self-modify yourself to go insane. (You can test yourself on data you've generated yourself, and where you know the correct answers, but that doesn't yet show that you'll process real-world data correctly.)

If I was an AI in such a situation, I'd make a modified copy of myself (or of the relevant modules) interfaced with a simulation environment with some physics-based puzzle to solve, such that it only gets a video feed and only has some simple controls (say, have it play Portal - the exact challenge is a bit irrelevant, just something that requires general intelligence). A modified AI that performs better (learns faster, comes up with better solutions) in a wide variety of simulated environments will probably also work better in the real world.

Even if the combinations of parameters that makes functional intelligence is very fragile, i.e. the search space has high-dimensionality and the "surface" is very jagged, it's still a search space that can be explored and mapped.

That's a bit hand-wavy, but enough to get me to suspect that an agent that can self-modify and run simulations of itself has a non-negligible chance of self-improving successfully (for a broad meaning of "successfully", that includes accidentally rewriting the utility function, as long as the resulting system is more powerful).

But the five year estimate doesn't strike me as unreasonable.

Meaning, a 1% chance of superhuman intelligence within 5 years, right?

Comment author: 20 December 2011 06:29:15PM 0 points [-]

Meaning, a 1% chance of superhuman intelligence within 5 years, right?

Sorry, I meant to say that it does not seem unreasonable to me that an AGI might take five years to self-improve. 1% does seem unreasonably low. I'm not sure what probability I would assign to "superhuman AGI in 5 years", but under say 40% seems quite low.

Comment author: 21 December 2011 12:37:49AM 3 points [-]

I've thought about scenarios of failed RSIs. My favorite is an idiot savant computer hacking AI that subsumes the entire Internet but has no conception of the real world. So we just power off, reformat and need to think carefully about how we make computers and how to control AI.

But I've really no concrete reason to expect this scenario to play out. I expect the nature of intelligence to throw us some more conceptual curve balls before we have an inkling of where we are headed and how to best steer the future.

Comment author: 19 December 2011 06:00:01PM 10 points [-]

10%: 2050 (I also think P=NP in that year.) 50%: 2062

+40% in 12 specific years? Now that's a bold distribution.

Comment author: 19 December 2011 10:58:05PM 2 points [-]

And he it would be even tighter on the left if P!=NP.

Comment author: 19 December 2011 09:13:10PM 3 points [-]

Is it plausible that fair-to-middling AI could be enough to break civilization? There are a lot of factors, especially whether civilization will become more fragile or more resilient as tech advances, but it does seem to me that profit-maximizing and status-maximizing AI have a lot of possibilities for trouble.

Comment author: 21 December 2011 03:28:49AM *  1 point [-]

How about attention-maximizing AI, e.g. a game that optimizes for addictiveness — for the amount of person-hours humans spend playing it?

Comment author: 22 December 2011 06:56:29PM 1 point [-]

I think that's less likely to break civilization than a status-maximizer or a money-maximizer-- there are a lot of people who don't want to get started with video games, and I think that an attention-maximizer would run into a lot of resistance as early adopters neglected their lives.

A clever attention maximizer which was aiming for the long run might not wreck civilization. I'm not as sure about status or money maximizers.

Brunner's The Jagged Orbit is about some issues with maximizers, including who might be most likely to develop a stupid maximizer.

Gur rkrphgvirf ng n crefbany jrncbaf pbzcnal unir n pbzchgre cebtenz gb znkvzvmr cebsvgf (cbffvoyl fnyrf-- V unira'g ernq gur obbx yngryl) va gur eryngviryl fubeg eha, erfhygvat va nqiregvfvat pnzcnvtaf juvpu penax hc cnenabvn gb gur cbvag jurer pvivyvmngvba vf jerpxrq.

Gurer'f n fbyhgvba juvpu vaibyirf gur pbzchgre vairagvat gvzr geniry, ohg V qba'g erzrzore gur qrgnvyf.

Comment author: 20 December 2011 10:56:48PM 6 points [-]

I do value these; please keep doing them!

Comment author: 23 December 2011 07:28:35PM 0 points [-]

That is very... refreshing.