I do not have the necessary education to evaluate state of the art AI research and to grasp associated fields that are required to make predictions about the nature of possible AI's capable of self-modification. I can only voice some doubts and questions.
For what it's worth, here are some thoughts on recursive self-improvement and risks from AI that I wrote for a comment on Facebook:
I do think that an expected utility-maximizer is the ideal in GAI. But, just like general purpose quantum computers, I believe that expected utility-maximizer's which - 1) find it instrumentally useful to undergo recursive self-improvement 2) find it instrumentally useful to take over the planet/universe to protect their goals - are, if at all feasible, the end-product of a long chain of previous AI designs with no quantum leaps in-between. That they are at all feasible is dependent on 1) how far from the human level intelligence hits diminishing returns 2) that intelligence is more useful than other kinds of resources in stumbling upon unknown unknowns in design space 3) that expected utility-maximizer's and their drives are not fundamentally dependent on the precision with which their utility-function is defined.
Here is an important question: Would Marilyn vos Savant (http://en.wikipedia.org/wiki/Marilyn_vos_Savant) be proportionally more likely to take over the world if she tried to than a 115 IQ individual?
Let me explain why I believe that the question is an important one. I believe that the question does shed light on the possibility of recursive self improvement and its economic feasibility.
An AI has to be able to estimate the expected utility of improving its own intelligence. And I think it is unlikely that any level of intelligence is capable of estimating the expected utility of a whole level above its own (where "a level above its own" is assumed to be similar to a boost in efficient cross-domain optimization power similar to that between humans and chimpanzees).
I think it is impossible for an AI to estimate the expected utility of the next level above its own, because 1) it can't possible know where the next level is located in design space 1b) how it can detect insights about it in solution space (because those insights are beyond a conceptual singularity (otherwise it wouldn't be a level above its own)) 2) how much resources it takes to stumble upon the next level 3) how long it takes to discover it.
Therefore any AI has to content to calculate the expected utility of the next small step towards the next level, the expected utility of small amplifications of its intelligence similar to the difference between an average human and that of Marylin vos Savant.
The reason for why an AI can't estimate the expected utility of the next level is that it is over its conceptual horizon, whereas small amplification are in sight. Small amplifications are subject to feedback from experimentation with altered designs and the use of guided evolution. Large amplifications require conceptual insights that are not readily available. No intelligence is able to easily verify conclusively the workings of another intelligence that is a level above its own without painstakingly acquiring resources, devising the necessary tools and building the required infrastructure.
Humans first had to invent science, bumble through the industrial revolution and develop computers to be able to prove modern mathematical problems. An AI would have to invent meta-science, maybe advanced nanotechnology, and other unknown tools and heuristics to be able to figure out how to create a trans-AI, an intelligence that could solve problems it couldn't solve itself.
Every level of intelligence has to prove the efficiency of its successor to estimate if it is rational to build it, if it is economical, if the resources that are necessary to build it should be allocated differently. This does demand considerable effort and therefore resources. It does demand great care and extensive simulations and being able to prove the correctness of the self-modification symbolically.
In any case, every level of intelligence has to acquire new resources given its current level of intelligence. It can't just self-improve to come up with faster and more efficient solutions. Self-improvement does demand resources. Therefore the AI is unable to profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.
For those reasons the AI has to answer the following questions,
There are many open questions here. It is not clear that most problems would be easier to solve given certain amounts of intelligence amplification. Since intelligence does not guarantee the discovery of unknown unknowns. Intelligence is mainly useful to adapt previous discoveries and solve well-defined problems. The next level of intelligence is by definition not well-defined. And even if it was the case that intelligence would guarantee to speed up the rate at which discoveries are made, it is not clear that the resources that are required to amplify intelligence are in proportion to its instrumental usefulness. It might be the case that many problems require exponentially more intelligence to make small steps towards a solution.
Yet there are still other questions, it is not clear that a lot of small steps of intelligence amplification eventually amount to a whole level. And as mentioned in the beginning, how dependent are AI's on the precision with which their goals are defined? If you tell an AI to create 10 paperclips, would it care to take over the universe to protect the paperclips from destruction? Would it care to create them economically or quickly? Would it care how to create 10 paperclips if those design parameters are not explicitly defined? I don't think so. More on that another time.
I can only voice some doubts and questions.
Just wanted to say, that I think it's great that you voice questions and doubts. Most folks who don't agree with the "party-line" on LW, or substantial amounts thereof, probably just leave.
I do not have the necessary education to evaluate state of the art AI research and to grasp associated fields that are required to make predictions about the nature of possible AI's capable of self-modification.
I don't have the necessary education either. But you can always make predictions, even if you know almost nothing about the topic in question. You just have to widen your confidence intervalls! :-)
[Click here to see a list of all interviews]
Homepage: cs.rutgers.edu/~mlittman/
Google Scholar: scholar.google.com/scholar?q=Michael+Littman
The Interview:
Michael Littman: A little background on me. I've been an academic in AI for not-quite 25 years. I work mainly on reinforcement learning, which I think is a key technology for human-level AI---understanding the algorithms behind motivated behavior. I've also worked a bit on topics in statistical natural language processing (like the first human-level crossword solving program). I carried out a similar sort of survey when I taught AI at Princeton in 2001 and got some interesting answers from my colleagues. I think the survey says more about the mental state of researchers than it does about the reality of the predictions.
In my case, my answers are colored by the fact that my group sometimes uses robots to demonstrate the learning algorithms we develop. We do that because we find that non-technical people find it easier to understand and appreciate the idea of a learning robot than pages of equations and graphs. But, after every demo, we get the same question: "Is this the first step toward Skynet?" It's a "have you stopped beating your wife" type of question, and I find that it stops all useful and interesting discussion about the research.
Anyhow, here goes:
Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?
Michael Littman:
10%: 2050 (I also think P=NP in that year.)
50%: 2062
90%: 2112
Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?
Michael Littman: epsilon, assuming you mean: P(human extinction caused by badly done AI | badly done AI)
I think complete human extinction is unlikely, but, if society as we know it collapses, it'll be because people are being stupid (not because machines are being smart).
Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?
Michael Littman: epsilon (essentially zero). I'm not sure exactly what constitutes intelligence, but I don't think it's something that can be turbocharged by introspection, even superhuman introspection. It involves experimenting with the world and seeing what works and what doesn't. The world, as they say, is its best model. Anything short of the real world is an approximation that is excellent for proposing possible solutions but not sufficient to evaluate them.
Q3-sub: P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
Michael Littman: Ditto.
Q3-sub: P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
Michael Littman: 1%. At least 5 years is enough for some experimentation.
Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?
Michael Littman: No, I don't think it's possible. I mean, seriously, humans aren't even provably friendly to us and we have thousands of years of practice negotiating with them.
Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?
Michael Littman: In terms of science risks (outside of human fundamentalism which is the only non-negligible risk I am aware of), I'm most afraid of high energy physics experiments, then biological agents, then, much lower, information technology related work like AI.
Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?
Michael Littman: I think people are currently hypersensitive. As I said, every time I do a demo of any AI ideas, no matter how innocuous, I am asked whether it is the first step toward Skynet. It's ridiculous. Given the current state of AI, these questions come from a simple lack of knowledge about what the systems are doing and what they are capable of. What society lacks is not a lack of awareness of risks but a lack of technical understanding to *evaluate* risks. It shouldn't just be the scientists assuring people everything is ok. People should have enough background to ask intelligent questions about the dangers and promise of new ideas.
Q7: Can you think of any milestone such that if it were ever reached you would expect human‐level machine intelligence to be developed within five years thereafter?
Michael Littman: Slightly subhuman intelligence? What we think of as human intelligence is layer upon layer of interacting subsystems. Most of these subsystems are complex and hard to get right. If we get them right, they will show very little improvement in the overall system, but will take us a step closer. The last 5 years before human intelligence is demonstrated by a machine will be pretty boring, akin to the 5 years between the ages of 12 to 17 in a human's development. Yes, there are milestones, but they will seem minor compared to first few years of rapid improvement.