= 27d567bc2d48d9f40e9ccc20ea370b15)
jsteinhardt comments on Q&A with Michael Littman on risks from AI - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (88)
I don't understand. XiXiDu's thinking was "if your assertion about humans was true, then we would expect to see these other things as well (i.e., other species being minimally fit for a task when they first start doing it); we therefore have a way of testing this hypothesis in a fairly convincing way, why don't we actually do that so that we can see if we're right or not?" That seems to me like the cornerstone of critical thought, or am I missing what you found objectionable?
That is a good suggestion and I endorse it. I have however been thinking about something else.
I suspected that people like cousin_it and wedrifid must base their assumption that human intelligence is close to the minimum level of efficiency (optimization power/resources used) on other evidence, e.g. that expert systems can factor numbers 10^180 times faster than humans can. I didn't think that the whole argument rests on the fact that humans didn't start to build a technological civilization right after they became mentally equipped to do so.
Don't get a wrong impression here, I agree that it is very unlikely that human intelligence is close to the optimum. But I also don't see that we have much reason to believe that it is close to the minimum. Further I believe that intelligence is largely overrated by some people on lesswrong and that conceptual revolutions, e.g. the place-value notation method of encoding numbers, wouldn't have been discovered much quicker by much more intelligent beings other than due to lucky circumstances. In other words, I think that the speed of discovery is not proportional to intelligence but rather that intelligence quickly hits diminishing returns (anecdotal evidence here includes that real world success doesn't really seem to scale with IQ points. Are people like Steve Jobs that smart? Could Terence Tao become the richest person if he wanted to? Are high karma people on lesswrong unusually successful?).
But I digress. My suggestion was to compare technological designs with evolutionary designs. For example animal echolocation with modern sonar, ant colony optimization algorithm with the actual success rate of ant behavior, energy efficiency and maneuverability of artificial flight with insect or bird flight...
If intelligence is a vastly superior optimization process compared to evolution then I suspect that any technological replications of evolutionary designs, that have been around for some time, should have been optimized to an extent that their efficiency vastly outperforms that of their natural counterparts. And from this we could then draw the conclusion that intelligence itself is unlikely to be an outlier but just like those other evolutionary designs very inefficient and subject to strong artificial amplification.
ETA: I believe that even sub-human narrow AI is an existential risk. So that I believe that lots of people here are hugely overconfident about a possible intelligence explosion doesn't really change that much with respect to risks from AI.
There's evidence that on the upper end higher IQ is inversely correlated with income but this may be due to people caring about non-monetary rewards (the correlation is positive at lower IQ levels and is positive at lower education levels but negatively correlated at higher education levels). I would not be surprised if there were an inverse correlation between high karma and success levels, since high karma may indicate procrastination and the like. If one looked at how high someone's average karma per a post is that might be a better metric to make this sort of point.
That occurred to me as well and seems a reasonable guess. But let me restate the question. Would Marilyn vos Savant be proportionally more likely to destroy the world if she tried to than a 115 IQ individual? I just don't see that and I still don't understand the hype about intelligence around here.
All it really takes is to speed up the rate of discovery immensely by having a vast number of sub-human narrow AI scientists research various dangerous technologies and stumble upon something unexpected or solve a problem that enables unfriendly humans to wreak havoc.
The main advantage of AI is that it can be applied in parallel to brute force a solution. But this doesn't imply that you can brute force problem solving and optimization power itself. Conceptual revolutions are not signposted so that one only has to follow a certain path or use certain heuristics to pick them up. They are scattered randomly across design space and their frequency is decreasing disproportionately with each optimization step.
I might very well be wrong and recursive self-improvement is a real and economic possibility. But I don't see there being enough evidence to take that possibility as seriously as some here do. It doesn't look like that intelligence is instrumentally useful beyond a certain point, a point that might well be close to human levels of intelligence. Which doesn't imply that another leap in intelligence, alike the one that happened since chimpanzees and humans split, isn't possible. But there is not much reason to believe that it can be reached by recursive self-improvement. It might very well take sheer luck because the level above ours is as hard to grasp for us than our level is for chimpanzees.
So even if it is economically useful to optimize our level of intelligence there is no evidence that it is possible to reach the next level other than by stumbling upon it. Lots of "ifs", lots of conjunctive reasoning is required to get from simple algorithms over self-improvement to superhuman intelligence.
I think that's the wrong question; it should read:
The difference in intelligence between Marylin vos Savant and a human with IQ 100 is very small in absolute terms.
I believe that the question is an important one. An AI has to be able to estimate the expected utility of improving its own intelligence and I think it is unlikely that any level of intelligence is capable of estimating the expected utility of a whole level above its own, because 1) it can't possible know where it is located in design space 1b) how it can detect insights in solution space 2) how much resources it takes 3) how long it takes. Therefore any AI has to calculate the expected utility of the next small step towards the next level, the expected utility of small amplifications of its intelligence similar to the difference between an average human and that of Marylin vos Savant. It has to ask 1) is the next step instrumentally useful 2) are resources spent on amplification better spent on 2b) pursuing other ways to gain power 2c) to pursue terminal goals directly given the current resources and intelligence.
I believe that those questions shed light on the possibility of recursive self improvement and its economic feasibility.
I think it's possible that an AI could at least roughly estimate where it's located in design space, how much ressources and how long it takes to increase its intelligence, but I'm happy to hear counter-arguments.
And it seems to me that the point of diminishing returns in increasing intelligence is very far away. I would do almost anything to gain, say, 30 IQ points, and that's nothing in absolute terms.
But I agree with you that these questions are important and that trusting my intuitions too much would be foolish.
I do not have the necessary education to evaluate state of the art AI research and to grasp associated fields that are required to make predictions about the nature of possible AI's capable of self-modification. I can only voice some doubts and questions.
For what it's worth, here are some thoughts on recursive self-improvement and risks from AI that I wrote for a comment on Facebook:
I do think that an expected utility-maximizer is the ideal in GAI. But, just like general purpose quantum computers, I believe that expected utility-maximizer's which - 1) find it instrumentally useful to undergo recursive self-improvement 2) find it instrumentally useful to take over the planet/universe to protect their goals - are, if at all feasible, the end-product of a long chain of previous AI designs with no quantum leaps in-between. That they are at all feasible is dependent on 1) how far from the human level intelligence hits diminishing returns 2) that intelligence is more useful than other kinds of resources in stumbling upon unknown unknowns in design space 3) that expected utility-maximizer's and their drives are not fundamentally dependent on the precision with which their utility-function is defined.
Here is an important question: Would Marilyn vos Savant (http://en.wikipedia.org/wiki/Marilyn_vos_Savant) be proportionally more likely to take over the world if she tried to than a 115 IQ individual?
Let me explain why I believe that the question is an important one. I believe that the question does shed light on the possibility of recursive self improvement and its economic feasibility.
An AI has to be able to estimate the expected utility of improving its own intelligence. And I think it is unlikely that any level of intelligence is capable of estimating the expected utility of a whole level above its own (where "a level above its own" is assumed to be similar to a boost in efficient cross-domain optimization power similar to that between humans and chimpanzees).
I think it is impossible for an AI to estimate the expected utility of the next level above its own, because 1) it can't possible know where the next level is located in design space 1b) how it can detect insights about it in solution space (because those insights are beyond a conceptual singularity (otherwise it wouldn't be a level above its own)) 2) how much resources it takes to stumble upon the next level 3) how long it takes to discover it.
Therefore any AI has to content to calculate the expected utility of the next small step towards the next level, the expected utility of small amplifications of its intelligence similar to the difference between an average human and that of Marylin vos Savant.
The reason for why an AI can't estimate the expected utility of the next level is that it is over its conceptual horizon, whereas small amplification are in sight. Small amplifications are subject to feedback from experimentation with altered designs and the use of guided evolution. Large amplifications require conceptual insights that are not readily available. No intelligence is able to easily verify conclusively the workings of another intelligence that is a level above its own without painstakingly acquiring resources, devising the necessary tools and building the required infrastructure.
Humans first had to invent science, bumble through the industrial revolution and develop computers to be able to prove modern mathematical problems. An AI would have to invent meta-science, maybe advanced nanotechnology, and other unknown tools and heuristics to be able to figure out how to create a trans-AI, an intelligence that could solve problems it couldn't solve itself.
Every level of intelligence has to prove the efficiency of its successor to estimate if it is rational to build it, if it is economical, if the resources that are necessary to build it should be allocated differently. This does demand considerable effort and therefore resources. It does demand great care and extensive simulations and being able to prove the correctness of the self-modification symbolically.
In any case, every level of intelligence has to acquire new resources given its current level of intelligence. It can't just self-improve to come up with faster and more efficient solutions. Self-improvement does demand resources. Therefore the AI is unable to profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.
For those reasons the AI has to answer the following questions,
There are many open questions here. It is not clear that most problems would be easier to solve given certain amounts of intelligence amplification. Since intelligence does not guarantee the discovery of unknown unknowns. Intelligence is mainly useful to adapt previous discoveries and solve well-defined problems. The next level of intelligence is by definition not well-defined. And even if it was the case that intelligence would guarantee to speed up the rate at which discoveries are made, it is not clear that the resources that are required to amplify intelligence are in proportion to its instrumental usefulness. It might be the case that many problems require exponentially more intelligence to make small steps towards a solution.
Yet there are still other questions, it is not clear that a lot of small steps of intelligence amplification eventually amount to a whole level. And as mentioned in the beginning, how dependent are AI's on the precision with which their goals are defined? If you tell an AI to create 10 paperclips, would it care to take over the universe to protect the paperclips from destruction? Would it care to create them economically or quickly? Would it care how to create 10 paperclips if those design parameters are not explicitly defined? I don't think so. More on that another time.
I don't know what that means. It's always possible to assign probabilities, even if you don't have a clue. And assigning utilities seems trivial, too. Let's say the AI thinks "Hm. Improving my intelligence will lead to world dominion, if a) vast intelligence improvement doesn't cost too much ressources, b) doesn't take too long, c) and if intelligence really is as useful as it seems to be, i.e. is more efficient at discovering "unknown unkowns in design space" than other processes (which seems to me tautological since intelligence is by definition optimization power divided by ressources used; but I could be wrong). Let me assign a 50% probability to each of these claims." (Or less, but it always can assign a probability and can therefore compute an expected utility. )
And so even if P(Gain World dominion|successful, vast intelligence-improvement) * P(succesful, vast intelligence-improvement) is small (and I think it's easily larger than 0.05) , the expected utility could be great nonetheless. If the AI is a maximizer, not a satisficer, it will try to take over the world.
The biggest problem that I have with recursive self-improvement is that it's not at all clear that intelligence is an easily "scalable" process. Some folks seem to think that intelligence is a relatively easy algorithm, that a few mathematical insights it will be possible to "grok" intelligence. But maybe you need many different modules and heuristics for general intelligence (just like the human brain) and there is no "one true and easy path". But I'm just guessing....
I agree that the EU of small improvements is easier to compute and that they are easier to implement. But if intelligence is an "scalable" process you can make those small improvements fairly rapidly and after 100 of those you should be pretty, friggin powerful.
Do you think the discovery of General Relativity was a well-defined problem? And what about writing of inspiring novels and creating beautiful art and music? Creativity is a subset of intelligence. There are no creative chimps.
What do you mean by intelligence?
And yes, I believe that people with very high IQ and advanced social skills (another instantiation of high intelligence; chimpanzees just don't have high social skills) are far more likely to take over the world than people with IQ 110, although it's still very unlikely.
My intuition tells me that they are. :-) If you give them the goal of creating 10 paperclips and nothing else, they will try everything to achieve this goal.
But Eliezer's arguments in the AI-Foom debate are far more convincing than mine, so my arguments tell you probably nothing new. The whole discussion is frustrating, because our (subconscious) intuitions seem to differ greatly, and there is little we can do about it. (Just like the debate between Hanson and Yudkowsky was pretty fruitless.) That doesn't mean that I don't want to participate in further discussions, but the probability of an agreement seems somewhat thin. I try to do my best :-)
I'm currently rereading the Sequences and I'm trying to summarize the various arguments and counter-arguments for the intelligence explosion, though that will take some time...
Just wanted to say, that I think it's great that you voice questions and doubts. Most folks who don't agree with the "party-line" on LW, or substantial amounts thereof, probably just leave.
I don't have the necessary education either. But you can always make predictions, even if you know almost nothing about the topic in question. You just have to widen your confidence intervalls! :-)
Yes, I was talking to people on Facebook who just "left".
The problem is that I find most of the predictions being made convincing, but only superficially so. There seem to be a lot of hidden assumptions.
If you were going to speed up a chimp brain a million times, would it quickly reach human-level intelligence? I don't think so. Why would it be different for a human-level intelligence trying to reach transhuman intelligence? It seems like a nice idea when formulated in English, but would it work?
Just because we understand Chessintelligence we do not understand Humanintelligence. As I see it, either there is a single theory of general intelligence and improving it is just a matter at throwing more resources at it or different levels are fundamentally different and you can't just interpolate Gointelligence from Chessintelligence...
Even if we assume that there is one complete theory of general intelligence. Once discovered, one just has to throw more resources at it. It might be able to incorporate all human knowledge, adapt it and find new patterns. But would it really be vastly superior to human society and their expert systems?
Take for example a Babylonian mathematician. If you traveled back in time and were to accelerate his thinking a million times, would he discover place-value notation to encode numbers in a few days? I doubt it. Even if he was to digest all the knowledge of his time in a few minutes, I just don't see him coming up with quantum physics after a short period of time.
That conceptual revolutions are just a matter of computational resources seems like pure speculation. If one were to speed up the whole Babylonian world and accelerate cultural evolution, obviously one would arrive quicker at some insights. But how much quicker? How much are many insights dependent on experiments, to yield empirical evidence, that can't be speed-up considerably? And what is the return? Is the payoff proportionally to the resources that are necessary?
Another problem is if one can improve intelligence itself apart from solving well-defined problems and making more accurate predictions on well-defined classes of problems. I don't think the discovery of unknown unknowns is subject to other heuristics than natural selection. Without goals, well-defined goals, terms like "optimization" have no meaning.
Without well-defined goals in form of a precise utility-function, I don't think it would be possible to maximize expected "utility". Concepts like "efficient", "economic" or "self-protection" all have a meaning that is inseparable with an agent's terminal goals. If you just tell it to maximize paperclips then this can be realized in an infinite number of ways that would all be rational given imprecise design and goal parameters. Undergoing to explosive recursive self-improvement, taking over the universe and filling it with paperclips, is just one outcome. Why would an arbitrary mind pulled from mind-design space care to do that? Why not just wait for paperclips to arise due to random fluctuations out of a state of chaos? That wouldn't be irrational. To have an AI take over the universe as fast as possible you would have to explicitly design it to do so.