..."if your assertion about humans was true, then we would expect to see these other things as well (i.e., other species being minimally fit for a task when they first start doing it); we therefore have a way of testing this hypothesis in a fairly convincing way,..."
That is a good suggestion and I endorse it. I have however been thinking about something else.
I suspected that people like cousin_it and wedrifid must base their assumption that human intelligence is close to the minimum level of efficiency (optimization power/resources used) on other evidence, e.g. that expert systems can factor numbers 10^180 times faster than humans can. I didn't think that the whole argument rests on the fact that humans didn't start to build a technological civilization right after they became mentally equipped to do so.
Don't get a wrong impression here, I agree that it is very unlikely that human intelligence is close to the optimum. But I also don't see that we have much reason to believe that it is close to the minimum. Further I believe that intelligence is largely overrated by some people on lesswrong and that conceptual revolutions, e.g. the place-value notation method of encoding numbers, wouldn't have been discovered much quicker by much more intelligent beings other than due to lucky circumstances. In other words, I think that the speed of discovery is not proportional to intelligence but rather that intelligence quickly hits diminishing returns (anecdotal evidence here includes that real world success doesn't really seem to scale with IQ points. Are people like Steve Jobs that smart? Could Terence Tao become the richest person if he wanted to? Are high karma people on lesswrong unusually successful?).
But I digress. My suggestion was to compare technological designs with evolutionary designs. For example animal echolocation with modern sonar, ant colony optimization algorithm with the actual success rate of ant behavior, energy efficiency and maneuverability of artificial flight with insect or bird flight...
If intelligence is a vastly superior optimization process compared to evolution then I suspect that any technological replications of evolutionary designs, that have been around for some time, should have been optimized to an extent that their efficiency vastly outperforms that of their natural counterparts. And from this we could then draw the conclusion that intelligence itself is unlikely to be an outlier but just like those other evolutionary designs very inefficient and subject to strong artificial amplification.
ETA: I believe that even sub-human narrow AI is an existential risk. So that I believe that lots of people here are hugely overconfident about a possible intelligence explosion doesn't really change that much with respect to risks from AI.
(anecdotal evidence here includes that real world success doesn't really seem to scale with IQ points. Are people like Steve Jobs that smart? Could Terence Tao become the richest person if he wanted to? Are high karma people on lesswrong unusually successful?).
There's evidence that on the upper end higher IQ is inversely correlated with income but this may be due to people caring about non-monetary rewards (the correlation is positive at lower IQ levels and is positive at lower education levels but negatively correlated at higher education levels). I w...
[Click here to see a list of all interviews]
Homepage: cs.rutgers.edu/~mlittman/
Google Scholar: scholar.google.com/scholar?q=Michael+Littman
The Interview:
Michael Littman: A little background on me. I've been an academic in AI for not-quite 25 years. I work mainly on reinforcement learning, which I think is a key technology for human-level AI---understanding the algorithms behind motivated behavior. I've also worked a bit on topics in statistical natural language processing (like the first human-level crossword solving program). I carried out a similar sort of survey when I taught AI at Princeton in 2001 and got some interesting answers from my colleagues. I think the survey says more about the mental state of researchers than it does about the reality of the predictions.
In my case, my answers are colored by the fact that my group sometimes uses robots to demonstrate the learning algorithms we develop. We do that because we find that non-technical people find it easier to understand and appreciate the idea of a learning robot than pages of equations and graphs. But, after every demo, we get the same question: "Is this the first step toward Skynet?" It's a "have you stopped beating your wife" type of question, and I find that it stops all useful and interesting discussion about the research.
Anyhow, here goes:
Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?
Michael Littman:
10%: 2050 (I also think P=NP in that year.)
50%: 2062
90%: 2112
Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?
Michael Littman: epsilon, assuming you mean: P(human extinction caused by badly done AI | badly done AI)
I think complete human extinction is unlikely, but, if society as we know it collapses, it'll be because people are being stupid (not because machines are being smart).
Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?
Michael Littman: epsilon (essentially zero). I'm not sure exactly what constitutes intelligence, but I don't think it's something that can be turbocharged by introspection, even superhuman introspection. It involves experimenting with the world and seeing what works and what doesn't. The world, as they say, is its best model. Anything short of the real world is an approximation that is excellent for proposing possible solutions but not sufficient to evaluate them.
Q3-sub: P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
Michael Littman: Ditto.
Q3-sub: P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
Michael Littman: 1%. At least 5 years is enough for some experimentation.
Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?
Michael Littman: No, I don't think it's possible. I mean, seriously, humans aren't even provably friendly to us and we have thousands of years of practice negotiating with them.
Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?
Michael Littman: In terms of science risks (outside of human fundamentalism which is the only non-negligible risk I am aware of), I'm most afraid of high energy physics experiments, then biological agents, then, much lower, information technology related work like AI.
Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?
Michael Littman: I think people are currently hypersensitive. As I said, every time I do a demo of any AI ideas, no matter how innocuous, I am asked whether it is the first step toward Skynet. It's ridiculous. Given the current state of AI, these questions come from a simple lack of knowledge about what the systems are doing and what they are capable of. What society lacks is not a lack of awareness of risks but a lack of technical understanding to *evaluate* risks. It shouldn't just be the scientists assuring people everything is ok. People should have enough background to ask intelligent questions about the dangers and promise of new ideas.
Q7: Can you think of any milestone such that if it were ever reached you would expect human‐level machine intelligence to be developed within five years thereafter?
Michael Littman: Slightly subhuman intelligence? What we think of as human intelligence is layer upon layer of interacting subsystems. Most of these subsystems are complex and hard to get right. If we get them right, they will show very little improvement in the overall system, but will take us a step closer. The last 5 years before human intelligence is demonstrated by a machine will be pretty boring, akin to the 5 years between the ages of 12 to 17 in a human's development. Yes, there are milestones, but they will seem minor compared to first few years of rapid improvement.