[Link] Better results by changing Bayes’ theorem

3 XiXiDu 09 March 2012 07:38PM

If it ever turns out that Bayes fails - receives systematically lower rewards on some problem, relative to a superior alternative, in virtue of its mere decisions - then Bayes has to go out the window.

-- Eliezer Yudkowsky, Newcomb's Problem and Regret of Rationality

Don't worry, we don't have to abandon Bayes’ theorem yet. But changing it slightly seems to be the winning Way given certain circumstances. See below:

In Peter Norvig’s talk The Unreasonable Effectiveness of Data, starting at 37:42, he describes a translation algorithm based on Bayes’ theorem. Pick the English word that has the highest posterior probability as the translation. No surprise here. Then at 38:16 he says something curious.

So this is all nice and theoretical and pure, but as well as being mathematically inclined, we are also realists. So we experimented some, and we found out that when you raise that first factor [in Bayes' theorem] to the 1.5 power, you get a better result.

In other words, if we change Bayes’ theorem (!) we get a better result. He goes on to explain

Link: johndcook.com/blog/2012/03/09/monkeying-with-bayes-theorem/

Peter Norvig - The Unreasonable Effectiveness of Data

How does real world expected utility maximization work?

12 XiXiDu 09 March 2012 11:20AM

I would like to ask for help on how to use expected utility maximization, in practice, to maximally achieve my goals.

As a real world example I would like to use the post 'Epistle to the New York Less Wrongians' by Eliezer Yudkowsky and his visit to New York.

How did Eliezer Yudkowsky compute that it would maximize his expected utility to visit New York?

It seems that the first thing he would have to do is to figure out what he really wants, his preferences1, right? The next step would be to formalize his preferences by describing it as a utility function and assign a certain number of utils2 to each member of the set, e.g. his own survival. This description would have to be precise enough to figure out what it would mean to maximize his utility function.

Now before he can continue he will first have to compute the expected utility of computing the expected utility of computing the expected utility of computing the expected utility3 ... and also compare it with alternative heuristics4.

He then has to figure out each and every possible action he might take, and study all of their logical implications, to learn about all possible world states he might achieve by those decisions, calculate the utility of each world state and the average utility of each action leading up to those various possible world states5.

To do so he has to figure out the probability of each world state. This further requires him to come up with a prior probability for each case and study all available data. For example, how likely it is to die in a plane crash, how long it would take to be cryonically suspended from where he is in case of a fatality, the crime rate and if aliens might abduct him (he might discount the last example, but then he would first have to figure out the right level of small probabilities that are considered too unlikely to be relevant for judgment and decision making).

I probably miss some technical details and got others wrong. But this shouldn't detract too much from my general request. Could you please explain how Less Wrong style rationality is to be applied practically? I would also be happy if you could point out some worked examples or suggest relevant literature. Thank you.

I also want to note that I am not the only one who doesn't know how to actually apply what is being discussed on Less Wrong in practice. From the comments:

You can’t believe in the implied invisible and remain even remotely sane. [...] (it) doesn’t just break down in some esoteric scenarios, but is utterly unworkable in the most basic situation. You can’t calculate shit, to put it bluntly.

None of these ideas are even remotely usable. The best you can do is to rely on fundamentally different methods and pretend they are really “approximations”. It’s complete handwaving.

Using high-level, explicit, reflective cognition is mostly useless, beyond the skill level of a decent programmer, physicist, or heck, someone who reads Cracked.

I can't help but agree.

P.S. If you really want to know how I feel about Less Wrong then read the post 'Ontological Therapy' by user:muflax.

 

1. What are "preferences" and how do you figure out what long-term goals are stable enough under real world influence to allow you to make time-consistent decisions?

2. How is utility grounded and how can it be consistently assigned to reflect your true preferences without having to rely on your intuition, i.e. pull a number out of thin air? Also, will the definition of utility keep changing as we make more observations? And how do you account for that possibility?

3. Where and how do you draw the line?

4. How do you account for model uncertainty?

5. Any finite list of actions maximizes infinitely many different quantities. So, how does utility become well-defined?

[Link] Personality change key to improving wellbeing

9 XiXiDu 06 March 2012 11:30AM

Is Personality Fixed? Personality Changes as Much as “Variable” Economic Factors and More Strongly Predicts Changes to Life Satisfaction,’ published in Social Indicators Research (doi: 10.1007/s11205-012-0006-z)

[...] small positive personality changes may lead to greater increases in happiness than earning more money, marrying, or gaining employment.

[...]

We found that our personalities can and do change over time – something that was considered improbable until now – and that these personality changes are strongly related to changes in our wellbeing.

[...]

Previous studies have shown that personality accounts for up to 35% of individual differences in life satisfaction, compared to just 4% for income, 4% for employment status and between 1% and 4% for marital status. However, because it was believed our personalities were fixed, policies to improve wellbeing have focused on these lower-impacting external factors.

[...]

“Fostering the conditions where personality growth occurs – such as through positive schooling, communities, and parenting - may be a more effective way of improving national wellbeing than GDP growth.”

[...]

Personality was measured using a well-validated personality questionnaire assessing five broad dimensions which cover the breadth of a person’s personality: openness-to-experiences, conscientiousness, extroversion, agreeableness and neuroticism. The researchers then looked at the extent to which personality changed and how these changes related to life satisfaction in comparison to external factors, such as changes to income, changes to employment and changes to marital status. They found that personality changes at least as much as these external factors and predicted about twice as much of changes to life satisfaction over the study period.

Link: manchester.ac.uk/aboutus/news/display/?id=8035

[Link] The emotional system (aka Type 1 thinking) might excel at complex decisions

7 XiXiDu 03 March 2012 07:05PM

For thousands of years, human beings have looked down on their emotions. We’ve seen them as primitive passions, the unfortunate legacy of our animal past. When we do stupid things – say, eating too much cake, or sleeping with the wrong person, or taking out a subprime mortgage – we usually blame our short-sighted feelings. People commit crimes of passion. There are no crimes of rationality.

This bias against feeling has led people to assume that reason is always best. When faced with a difficult dilemma, most of us believe that it’s best to carefully assess our options and spend a few moments consciously deliberating the information. Then, we should choose the alternative that best fits our preferences. This is how we maximize utility; rationality is our Promethean gift.

[...] it’s only in the last few years that researchers have demonstrated that the emotional system (aka Type 1 thinking) might excel at complex decisions, or those involving lots of variables.

[...]

The latest demonstration of this effect comes from the lab of Michael Pham at Columbia Business School. The study involved asking undergraduates to make predictions about eight different outcomes, from the Democratic presidential primary of 2008 to the finalists of American Idol. They forecast the Dow Jones and picked the winner of the BCS championship game. They even made predictions about the weather.

Here’s the strange part: although these predictions concerned a vast range of events, the results were consistent across every trial: people who were more likely to trust their feelings were also more likely to accurately predict the outcome. [...]

Consider the results from the American Idol quiz: while high-trust-in-feelings subjects correctly predicted the winner 41 percent of the time, those who distrusted their emotions were only right 24 percent of the time. The same lesson applied to the stock market, that classic example of a random walk: those emotional souls made predictions that were 25 percent more accurate than those who aspired to Spock-like cognition.

[...] the unconscious brain is able to process vast amounts of information in parallel, thus allowing it to analyze large data sets without getting overwhelmed. (Human reason, in contrast, has a very strict bottleneck and can only process about four bits of data at any given moment.)

[...] how do we gain access to all this analysis [...]

[...] emotions come in handy. Every feeling is like a summary of data, a quick encapsulation of all the information processing that we don’t have access to. (As Pham puts it, emotions are like a “privileged window” into the subterranean mind.) When it comes to making predictions about complex events, this extra information is often essential. It represents the difference between an informed guess and random chance.

[...] for example, that you’re given lots of information about how twenty different stocks have performed over a period of time.

[...] if you’re asked which stocks trigger the best feelings [...] you will suddenly be able to identify the best stocks [...] your feelings will “reveal a remarkable degree of sensitivity” to the actual performance of all of the different securities.

But this doesn’t meant we can simply rely on every fleeting whim [...] only benefit from the emotional oracle effect when they had some knowledge of the subject. If they weren’t following [...] then their feelings weren’t helpful predictors [...]

[...] our emotions [...] are imperfect oracles [...] a strong emotion is a reminder that, even when we think we know nothing, our brain knows something.

Link: wired.com/wiredscience/2012/03/are-emotions-prophetic/

Study: business.illinois.edu/ba/seminars/2010/pham_paper2.pdf

Q&A with experts on risks from AI #4

14 XiXiDu 19 January 2012 04:29PM

[Click here to see a list of all interviews]

Professor Michael G. Dyer is an author of over 100 publications, including In-Depth Understanding, MIT Press, 1983. He serves on the editorial board of the journals: Applied Intelligence, Connection Science, Knowledge-Based Systems, International Journal of Expert Systems, and Cognitive Systems Research. His research interests are centered around semantic processing of natural language, through symbolic, connectionist, and evolutionary techniques. [Homepage]

Dr. John Tromp is interested in Board Games and Artificial Intelligence, Algorithms, Complexity, Algorithmic Information Theory, Distributed Computing, Computational biology. His recent research has focused on the Combinatorics of Go, specifically counting the number of legal positions. [Homepage]

Dr. Kevin Korb both developed and taught the following subjects at Monash University: Machine Learning, Bayesian Reasoning, Causal Reasoning, The Computer Industry: historical, social and professional issues, Research Methods, Bayesian Models, Causal Discovery, Epistemology of Computer Simulation, The Art of Causal. [Curriculum vitae] [Bayesian Artificial Intelligence]

Dr. Leo Pape is a postdoc in Jürgen Schmidhuber's group at IDSIA (Dalle Molle Institute for Artificial Intelligence). He is interested in artificial curiosity, chaos, metalearning, music, nonlinearity, order, philosophy of science, predictability, recurrent neural networks, reinforcement learning, robotics, science of metaphysics, sequence learning, transcendental idealism, unifying principles. [Homepage] [Publications]

Professor Peter Gacs is interested in Fault-tolerant cellular automata, algorithmic information theory, computational complexity theory, quantum information theory. [Homepage]

Professor Donald Loveland does focus his research on automated theorem proving, logic programming, knowledge evaluation, expert systems, test-and-treatment problem. [Curriculum vitae]

Eray Ozkural is a computer scientist whose research interests are mainly in parallel computing, data mining, artificial intelligence, information theory, and computer architecture. He has an Msc. and is trying to complete a long overdue PhD in his field. He also has a keen interest in philosophical foundations of artificial intelligence. With regards to AI, his current goal is to complete an AI system based on the Alpha architecture of Solomonoff. His most recent work (http://arxiv.org/abs/1107.2788) discusses axiomatization of AI.

Dr. Laurent Orseau is mainly interested in Artificial General Intelligence, which overall goal is the grand goal of AI: building an intelligent, autonomous machine. [Homepage] [Publications] [Self-Modification and Mortality in Artificial Agents]

Richard Loosemore is currently a lecturer in the Department of Mathematical and Physical Sciences at Wells College, Aurora NY, USA. Loosemore's principle expertise is in the field known as Artificial General Intelligence, which seeks a return to the original roots of AI (the construction of complete, human-level thinking systems). Unlike many AGI researchers, his approach is as much about psychology as traditional AI, because he believes that the complex-system nature of thinking systems make it almost impossible to build a safe and functioning AGI unless its design is as close as possible to the design of the human cognitive system. [Homepage]

Monica Anderson has been interested in the quest for computer based cognition since college, and ever since then has sought out positions with startup companies that have used cutting-edge technologies that have been labeled as "AI". However, those that worked well, such as expert systems, have clearly been of the "Weak AI" variety. In 2001 she moved from using AI techiques as a programmer to trying to advance the field of "Strong AI" as a researcher. She is the founder of Syntience Inc., which was established to manage funding for her exploration of this field. She has a Master's degree in Computer Science from Linköping University in Sweden. She created three expert systems for Cisco Systems for product configuration verification; She has co-designed systems to automatically classify documents by content; She has (co-)designed and/or (co-)written LISP interpreters, debuggers, chat systems, OCR output parsers, visualization tools, operating system kernels, MIDI control real-time systems for music, virtual worlds, and peer-to-peer distributed database systems. She was Manager of Systems Support for Schlumberger Palo Alto Research. She has worked with robotics, industrial control, marine, and other kinds of embedded systems. She has worked on improving the quality of web searches for Google. She wrote a Genetic Algorithm which successfully generated solutions for the Set Coverage Problem (which has been shown to be NP-hard) around 1994. She has used more than a dozen programming languages professionally and designed or co-designed at least four programming languages, large or small. English is her third human language out of four or five. [More]

The Interview (New Questions)

Peter Gacs: I will try to answer your questions, but probably not all, and with some disclaimers.

As another disclaimer: the questions, and the website lesswrong.com that I glanced at, seem to be influenced by Raymond Kurzweil's books.  I have not read those books, though of course, I heard about them in conversations, and have seen some reviews.  I do not promise never to read them, but waiting for this would delay my answers indefinitely.

Laurent Orseau: Keep in mind that the thoughts expressed here reflect my state of mind and my knowledge at the time of the writing, and may significantly differ after further discussions, readings and thoughts. I have no definite idea about any of the given questions.

Q1: Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans at science, mathematics, engineering and programming?

Kevin Korb: 2050/2200/2500

The assumptions, by the way, are unrealistic. There will be disruptions.

John Tromp: I believe that, in my lifetime, computers will only be proficient at well-defined and specialized tasks. Success in the above disciplines requires too much real-world understanding and social interaction. I will not even attempt projections beyond my lifetime (let's say beyond 40 years).

Michael G. Dyer: See Ray Kurzweil's book:  The Singularity Is Near.

As I recall, he thinks it will occur before mid-century.

I think he is off by at least an additional 50 years (but I think we'll have as manypersonal robots as cars by 2100.)

One must also distinguish between the first breakthrough of a technology vs. that breakthrough becoming cheap enough to be commonplace, so I won't give you any percentages.  (Several decades passed between the first cell phone and billions of people having cell phones.)

Peter Gacs: I cannot calibrate my answer as exactly as the percentages require, so I will
just concentrate on the 90%.  The question is a common one, but in my opinion
history will not answer it in this form.  Machines do not develop in direct
competition of human capabilities, but rather in attempts to enhance and
complement them.  If they still become better at certain tasks, this is a side
effect.  But as a side effect, it will indeed happen that more and more tasks
that we proudly claim to be creative in a human way, will be taken over by
computer systems.  Given that the promise of artificial intelligence is by now
50 years old, I am very cautious with numbers, and will say that at least 80
more years are needed before jokes about the stupidity of machines will become
outdated.

Eray Ozkural: 2025/2030/2045.

Assuming that we have the right program in 2035 by 100% probability, it could still take about 10 years to train it adequately, even though we might find that our programs by then learn much faster than humans. I anticipate that the most expensive part of developing an AI will be training, although we tend to assume that after we bring it up to primary school level, i.e. it can read and write, it would be able to learn much on its own. I optimistically estimated that it would take $10 million dollars and 10 years to train an AI in basic science. Extending that to cover all four of science, mathematics, engineering and programming could take even longer. It takes a human, arguably 15-20 years of training to be a good programmer, and very few humans can program well after that much educational effort and expense.

Laurent Orseau:

10%: 2017
50%: 2032
90%: 2100

With a quite high uncertainty though.
My current estimate is that (I hope) we will know we have built a core AGI by 2025, but a lot of both research and engineering work and time (and learning for the AGI) will be required for the AGI to reach human level in most domains, up to 20 years in the worst case I speculate and 5 years at least, considering that a lot of people will probably be working on it at that time. That is, if we really want to make it human-like.

Richard Loosemore: 2015 - 2020 - 2025

Monica Anderson:

10%  2020
50%  2026
90%  2034

These are all Reductionist sciences. I assume the question is whether we'll have machines capable of performing Reduction in these fields. If working on pre-reduced problems, where we already have determined which Models (formulas, equations, etc) to use and know the values of all input variables, then we already have Mathematica. But here the Reduction was done by a human so Mathematica is not AI.

AIs would be useful for more everyday things, such as (truly) Understanding human languages years before they Understand enough to learn the Sciences and can perform full-blown Reduction. This is a much easier task, but is still AI-Complete. I think the chance we'll see a program truly Understand a human language at the level of a 14-year old teenager is

10% 2014
50% 2018
90% 2022

Such an AI would be worth hundreds of billions of dollars and makes a worthy near-term research goal. It could help us radically speed up research in all areas by allowing for vastly better text-based information filtering and gathering capabilities, perfect voice based input, perfect translation, etc.

Q2: Once we build AI that is roughly as good as humans at science, mathematics, engineering and programming, how much more difficult will it be for humans and/or AIs to build an AI which is substantially better at those activities than humans?

Kevin Korb: It depends upon how AGI is achieved. If it's through design breakthroughs in AI architecture, then the Singularity will follow. If it's through mimicking nanorecordings, then no Singularity is implied and may not occur at all.

John Tromp: Not much. I would guess something on the order of a decade or two.

Michael G. Dyer: Machines and many specific algorithms are already substantially better at their tasks than humans.
(What human can compete with a relational database?, or with a Bayesian reasoner? or with scheduler?, or with an intersection-search mechanism like WATSON? etc.)

For dominance over humans, machines have to first acquire the ability to understand human language and to have thoughts in the way humans have thoughts.   Even though the WATSON program is impressive, it does NOT know what a word actually means (in the sense of being able to answer the question:  "How does the meaning of the word "walk"  differ from the meaning of the word "dance", physically, emotionally, cognitively, socially?"

It's much easier to get computers to beat humans at technical tasks (such as sci, math, eng. prog.) but humans are vastly superior at understanding language, which makes humans the master of the planet.  So the real question is:  At what point will computers understand natural language as well as humans?

Peter Gacs: This is also hard to quantify, since in some areas machines will still be behind, while in others they will already be substantially better: in my opinion, this is already the case.  If I still need to give a number, I say 30 years.

Eray Ozkural: I expect that by the time such a knowledgeable AI is developed, it will already be thinking and learning faster than an average human. Therefore, I think, simply by virtue of continuing miniaturization of computer architecture, or other technological developments that increase our computational resources (e.g., cheaper energy technologies such as fusion), a general-purpose AI could vastly transcend human-level intelligence.

Laurent Orseau: Wild guess: It will follow Moore's law (see below).

Richard Loosemore: Very little difficulty. I expect it to happen immediately after the first achievement, because at the very least we could simply increase the clock speed in relevant areas. It does depend exactly how you measure "better", though.

Monica Anderson: What does "better" mean? If we believe, as many do, that Intelligence is for Prediction, and that the best measure of our intelligence is whether we can predict the future in complex domains, then we can interpret the given question as "when can an AI significantly outpredict a human in their mundane everyday environment".

For any reasonable definition of "significant", the answer is "never". The world is too complex to be predictable. All intelligences are "best-effort" systems where we do as best we can and learn from our mistakes when we fail, for fail we must. Human intelligences have evolved to the level they have because it is a reasonable level for superior survival chances in the environments in which we've evolved. More processing power, faster machines, etc.  do not necessarily translate into an improved ability to predict the environment, especially if we add AIs to this environment. A larger number of competent agents like AIs will make the domain even MORE complex, leading to LOWER predictability. For more about this, see http://hplusmagazine.com/2010/12/15/problem-solved-unfriendly-ai.

Improved ability to handle Models (creating a "super Mathematica") is of limited utility for the purpose of making longer-term predictions. Chains of Reductionist Models attempting to predict the future tend to look like Rube Goldberg machines and are very likely to fail, and to fail spectacularly (which is what Brittleness is all about).

Computers will not get better at Reduction (the main skill required for Science, Mathematics, Engineering, and Programming) until they gather a lot of experience of the real world. For instance, a programming task is 1% about Understanding programming and 99% about Understanding the complex reality expressed in the spec of the program. This can only be improved by Understanding reality better, which is a slow process with the limitations described above. For an introduction to this topic, see my article "Reduction Considered Harmful" at http://hplusmagazine.com/2011/03/31/reduction-considered-harmful.

The "Problem with Reduction" is actually "The Frame Problem" as described by John McCarthy and Pat Hayes, viewed from a different angle. It is not a problem that AI research can continue to ignore, which is what we've done for decades. It will not go away. The only approach that works is to sidestep the issue of continuous Model update by not using Models. AIs must use nothing but Model Free Methods since these work without performing Reduction (to Models) and hence can be used to IMPLEMENT automatic Reduction.

Q3: Do you ever expect artificial intelligence to overwhelmingly outperform humans at typical academic research, in the way that they may soon overwhelmingly outperform humans at trivia contests, or do you expect that humans will always play an important role in scientific progress?

Kevin Korb: They will overwhelmingly outperform if and only if we achieve artificial general intelligence through human understanding of intelligence or through artificial understanding of intelligence (vs nanomeasurements).

John Tromp: Like I said, not in my lifetime, and projections beyond that are somewhat meaningless I think.

Michael G. Dyer: Regarding trivia contests, WATSON's performance fools people into thinking that the language problem has been solved, but the WATSON program does not understand language.   It does an intersection search across text so, for example, if it knows that the answer category is human and that a clue is "Kitty Hawk" then it can do an intersection search and come up with Wright Brothers.  The question can sound complicated but WATSON can avoid comprehending the question and just return the best intersection that fits the answer category.  It can treat each sentence as a bag of words/phrases.  WATSON cannot read a child's story and answer questions about what the characters wanted and why they did what they did.

Peter Gacs: Absolutely, but the contest will never be direct, and therefore the victory will never have to be acknowledged.  Whatever tasks the machines are taking over, will always be considered as tasks that are just not worthy of humans.

Eray Ozkural: Yes. In Ray Solomonoff's paper titled "The time scale of artificial intelligence: Reflections on social effects", he predicted that it will be possible to build an AI that is as intelligent as many times the entire computer science community (AI Milestone F). He predicted that it would take a short time to go from a human-level AI, to such a vastly intelligent AI that would overwhelmingly outperform not only individual humans, but the entire computer science community. This is called the "infinity point" hypothesis, and it was the first scientific formulation of singularity (1985). He formalized the feedback loop by which an AI could increase its own intelligence by working on miniaturization of computer architectures, e.g, Moore's law. The idea is that by being smarter than humans, the AI would accelerate Moore's law, theoretically achieving infinite intelligence in a short, finite time, depending on the initial investment.

However, of course, infinite intelligence is impossible due to physical limits. Unaided Moore's law can only continue up to physical limits of computation which would be reached by 2060's if current rate of progress continued, and needless to mention those limits are sort of impossible to achieve (since they might involve processes that are a bit like blackholes). However, imagine this, the AI could design fusion reactors using the H3 on Moon and energy-efficient processors to achieve large amounts of computation. There could be alternative ways to obtain extremely fast supercomputers, and so forth, Solomonoff's hypothesis could be extended to deal with all sorts of technological advances, for instance a self-improving AI could improve its own code, which designs like Goedel Machine and Solomonoff's Alpha are supposed to accomplish. Therefore, ultimately, such AI's would help improve computer architecture, artificial intelligence, electronics, aerospace, energy, communication technologies, all of which would help build AI's that are perhaps hundreds of thousands of times smarter than individual humans, or perhaps much smarter than the entire humanity as Ray Kurzweil predicts, not just particular scientific communities like the computer science community,

Laurent Orseau: "Always" is a very strong word. So probably not for that last part.
I give 100% chances for an AI to vastly outperform humans in some domains (which we already have algorithms for, like calculus and chess of course), 50% in many domains, and 10% in all domains. Humans have some good old genetic biases that might be hard to challenge.
But how much better it will be is still very unclear, mostly due to NP-hardness, Legg's prediction hardness results and related no-free-lunch problems, where progress might only be gained through more computing power.

The AGI might have significantly different hot research topics than humans, so I don't think we will lose our philosophers that fast. And good philosophy can only be done with good science.

Also, machines are better at chess and other games than me, but that doesn't prevent me from playing.

Richard Loosemore: Yes. Except for one thing. Humans will be able to (and some will choose to) augment their own intellectual capacity to the same level as the AIs. In that case, your question gets a little blurred.

Monica Anderson: I don't believe AI will even reliably "overwhelmingly outperform" humans at trivia contests until they fully Understand language. Language Understanding computers will be a great help, but the overwhelming outperformance in Reduction-related tasks is unlikely to happen. Reduction is very difficult.

Q4: What probability do you assign to the possibility of an AI with initially (professional) human-level competence at general reasoning (including science, mathematics, engineering and programming) to self-modify its way up to vastly superhuman capabilities within a matter of hours/days/< 5 years?

Kevin Korb: If through nanorecording: approx 0%. Otherwise, the speed/acceleration at which AGIs improve themselves is hard to guess at.

John Tromp: I expect such modification will require plenty of real-life interaction.

  • hours:      10^-9
  • days:       10^-6
  • <5 years : 10^-1

Michael G. Dyer: -

Peter Gacs: This question presupposes a particular sci-fi scenario that I do not believe in.

Eray Ozkural: In 5 years, without doing anything, it would already be faster than a human simply by running on a faster computer. If Moore's law continued by then, it would be 20-30 times faster than a human. But if you mean by "vastly" a difference of thousand times faster, I give it a probability of only 10% because there might be other kinds of bottlenecks involved (mostly physical). There is also another problem with Solomonoff's hypothesis, which Kurzweil generalized, that we are gladly omitting. An exponential increase in computational speed may only amount to a linear increase in intelligence. It at least corresponds only to a linear increase in the algorithmic complexity of solutions that can be found by any AGI, which is a well known fact, and cannot be worked around by simple shortcuts. If solution complexity is the best measure of intelligence, then, getting much more intelligent is not so easy (take this with a grain of salt, though, and please contrast it with the AIQ idea).

Laurent Orseau: I think the answer to your question is similar to the answer to:
Suppose we suddenly find a way to copy and emulate a whole human brain on a computer; How long would it take *us* to make it vastly better than it is right now?
My guess is that we will make relatively slow progress. This progress can get faster with time, but I don't expect any sudden explosion. Optimizing the software sounds a very hard task, if that is even possible: if there were an easy way to modify the software, it is probable that natural selection would have found it by now. Optimizing the hardware should then follows Moore's law, at least for some time.
That said, the digital world might allow for some possibilities that might be more difficult in a real brain, like copy/paste or memory extension (although that one is debatable).

I don't even know if "vastly superhuman" capabilities is something that is even possible. That sounds very nice (in the best scenario) but is a bit dubious. Either Moore's law will go on forever, or it will stop at some point. How much faster than a human can a computer compute, taking thermodynamics into account?

So, before it really becomes much more intelligent/powerful than humans, it should take some time.
But we may need to get prepared for otherwise, just in case.

Richard Loosemore: Depending on the circumstances (which means, this will not be possible if the AI is built using dumb techniques) the answer is: near certainty.

Monica Anderson: 0.00% . Reasoning is useless without Understanding because if you don't Understand (the problem domain), then you have nothing to reason about. Symbols in logic have to be anchored in general Understanding of the problem domain we're trying to reason about.

Q5: How important is it to research risks associated with artificial intelligence that is good enough at general reasoning (including science, mathematics, engineering and programming) to be capable of radical self-modification, before attempting to build one?

Kevin Korb: It is the key issue in the ethics of AI. Without a good case to make, the research may need to cease. To be sure, one aspect of a good case may well be that unethical projects are underway and likely to succeed. Per my answers above, I do not currently believe anything of the kind. No project is near to success.

John Tromp: Its importance grows with the extent to which we allow computers control over critical industrial/medical/economic processes, infrastructure, etc. As long as their role is limited to assisting humans in control, there appears to be little risk.

Michael G. Dyer: A robot that does not self-replicate is probably not very dangerous (leaving out robots for warfare).
A robot that wants to make multiple copies of itself would be dangerous (because it could undergo a rapid form of Lamarckian evolution.  There are two type of replication:   factory replication and factory division via the creation of a new factory.   In social insects this is the difference between the queen laying new eggs and a hive splitting up to go build new hive structures at a new site.

Assuming that humans remain in control of the energy and resources to a robot-producing factory, then factory replication could be shut down.  Robots smart enough to go build a new factory and maintain control over the needed resources would pose the more serious problem.  As robots are designed (and design themselves) to follow their own goals (for their own self-survival, especially in outer space) then those goals will come into conflict with those of humans.   Asimov's laws are too weak to protect humans and as robots design new versions of themselves then they will eliminate those laws anyway.

Monica Anderson: Not very important. Radical self-modification cannot be undertaken by anyone (including AIs) without Understanding of what would make a better Understander. While it is possible that an AI could be helpful in this research I believe the advances in this area would be small, slow to arrive, and easy to control, hitting various brick walls of radically diminishing returns that easily dis-compensates advances of all kinds including Moore's Law.

We already use computers to design faster, better, logically larger and physically smaller computers. This has nothing to do with AI since the improvements come from Understanding about the problem domain – computer design – that is performed by humans. Greater capability in a computer translates to very small advances in Reductive capability. Yes, Understanding machines may be able to eventually Understand Understanding to the point of creating a better Understander. This is a long ways off; Understanding Understanding is uncommon even among humans. But even then, the unpredictability of our Mundane reality is what limits he advantage any intelligent agent might have.

Q5-old: How important is it to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to build AI that is good enough at general reasoning (including science, mathematics, engineering and programming) to undergo radical self-modification?

Peter Gacs: This is an impossible task.  "AI" is not a separate development that can be regulated the way that governments regulate research over infectious bacteria to make sure they do not escape the laboratory.  Day for day, we are yielding decision power to smart machines, since we draw---sometimes competitive---advantage from this.  Emphasizing that the process is very gradual, I still constructed a parable that illustrates the process via a quick and catastrophic denuement.

Thinking it out almost fourty years ago, I assumed that the nuclear superpowers, the Soviet Union and the USA, would live on till the age of very smart machines. So, at some day, for whatever reason, World War 3 breaks out between these superpowers.  Both governments consult their advanced computer systems on how to proceed, and both sides get analogous answers.  The Soviet computer says: the first bomb must be dropped on the Kremlin; in the US, the advice is to drop the first bomb on the Pentagon.  The Americans still retain enough common sense to ignore the advice; but Soviets are more disciplined, and obey their machine. After the war plays out, the Soviet side wins, since the computer advice was correct on both sides. (And from then on, machines rule...)

Eray Ozkural: A sense of benevolence or universal ethics/morality would only be required if the said AI is also an intelligent agent that would have to interact socially with humans. There is no reason for a general-purpose AI to be an intelligent agent, which is an abstraction of animal, i.e., as commonly known as an "animat" since early cyberneticists. Instead, the God-level intelligence could be an ordinary computer that solves scientific problems on demand. There is no reason for it to control robotic hardware or act on its own, or act like a human or an animal. It could be a general-purpose expert system of some sort, just another computer program, but one that is extremely useful. Ray Solomonoff wrote this about human-like behavior in his paper presented at the 2006 Dartmouth Artificial Intelligence conference (50th year anniversary) titled"Machine Learning - Past and Future", which you can download from his website:

http://world.std.com/~rjs/dart.pdf

"To start, I’d like to define the scope of my interest in A.I. I am not particularly interested in simulating human behavior. I am interested in creating a machine that can work very difficult problems much better and/or faster than humans can – and this machine should be embodied in a technology to which Moore’s Law applies. I would like it to give a better understanding of the relation of quantum mechanics to general relativity. I would like it to discover cures for cancer and AIDS. I would like it to find some very good high temperature superconductors. I would not be disappointed if it were unable to pass itself off as a rock star."

That is, if you constrain the subject to a non-autonomous, scientific AI, I don't think you'll have to deal with human concepts like "friendly" at all. Without even mentioning how difficult it might be to teach any common sense term to an AI. For that, you would presumably need to imitate the way humans act and experience.

However, to solve the problems in science and engineering that you mention, a robotic body, or a fully autonomous, intelligent agent, is not needed at all. Therefore, I think it is not very important to work on friendliness for that purpose. Also, one person's friend is another's enemy. Do we really want to introduce more chaos to our society?

Laurent Orseau: It is quite dubious that "provably friendly" is something that is possible.
A provably friendly AI is a dead AI, just like a provably friendly human is a dead human, at least because of how humans would use/teach it, and there are bad guys who would love to use such a nice tool.
The safest "AI" system that I can think of is a Q/A system that is *not* allowed to ask questions (i.e. to do actions). But then it cannot learn autonomously and may not get as smart as we'd like, at least in reasonable time; I think it would be quite similar to a TSP solver: its "intelligence" would be tightly linked to its CPU speed.

"Provably epsilon-friendly" (with epsilon << 1 probability that it might not be always friendly) is probably a more adequate notion, but I'm still unsure this is possible to get either, though maybe under some constraints we might get something.

That said, I think this problem is quite important, as there is still a non-negligible possibility that an AGI gets much more *power* (no need for vastly more intelligence) than humanity, even without being more intelligent. An AGI could travel at the speed of information transfer (so, light speed) and is virtually immortal by restoring from backups and creating copies of itself. It could send emails on behalf of anyone, and could crack high security sites with as much social engineering as we do. As it would be very hard to put in jail or to annihilate, it would feel quite safe (for its own life) to do whatever it takes to achieve its goals.
Regarding power and morality (i.e. what are good goals), here is a question: Suppose you are going for a long walk in the woods in a low populated country, on your own. In the middle of the walk, some big guy pops out of nowhere and comes to talk to you. He is ugly, dirty, smells horribly bad, and never stops talking. He gets really annoying, poking you and saying nasty things, and it's getting worse and worse. You really can't stand it anymore. You run, you go back and forth, you shout at him, you vainly try to reason him but you can't get rid of him. He just follows you everywhere. You don't really want to start a fight as he looks much stronger than you are. Alas, it will take you some 5 more hours to get back to your car and nobody else is in the woods. But In your pocket you have an incredible device: A small box with a single button that can make everything you wish simply disappear instantly. No blood, no pain, no scream, no trace, no witness, no legal problem, 100% certified. At some instant the guy would be here, the next instant he would not, having simply vanished. As simple as that. You don't know what happens to the disappeared person. Maybe he dies, maybe he gets teleported somewhere, or reincarnated or whatever. You know that nobody knows this guy, so nobody can miss him or even look for him. You try to explain to him what this box is, you threaten him to press the button but he does not care. And he's getting so, so annoying, that you can't refrain to scream. Then you stare at the button... Will you press it?
My guess is that most people would like to say no, because culture and law say it's bad, but the truth may be that most of them would be highly tempted if facing such a situation. But if they had a gun or a saber instead of a button, the answer would probably be a straighter no (note that a weapon injury is much like a death sentence in the woods). The definition of morality might depend on the power you have.

But, hopefully, we will be sufficiently smart to put a number of safety measures and perform a lot of testing under stressful conditions before launching it in the wild.

Richard Loosemore: Absolutely essential. Having said that, the task of making it "provably" friendly is not as difficult as portrayed by organizations (SIAI, FHI) that have a monomaniacal dedication to AI techniques that make it impossible. So in other words: essential, but not a difficult task at all.

Q6: What probability do you assign to the possibility of human extinction within 100 years as a result of AI capable of self-modification (that is not provably non-dangerous, if that is even possible)? P(human extinction by AI | AI capable of self-modification and not provably non-dangerous is created)

Kevin Korb: This question is poorly phrased.

You should ask relative to a time frame. After all, the probability of human extinction sometime or other is 1. (Note by XiXiDu: I added "within 100 years" to the question after I received his answers.)

"Provably" is also problematic. Outside of mathematics, little is provable.

My generic answer is that we have every prospect of building an AI that behaves reasonably vis-a-vis humans, should we be able to build one at all. We should, of course, take up those prospects and make sure we do a good job rather than a bad one.

John Tromp: The ability of humans to speed up their own extinction will, I expect, not be matched any time soon by machine, again not in my lifetime.

Michael G. Dyer: Loss of human dominance is a foregone conclusion (100% for loss of dominance). Every alien civilization (including humans) that survives its own annihilation (via nuclear, molecular and nano technologies) will at some point figure out how to produce synthetic forms of its own intelligence.   These synthetics beings are necessary for space travel (because there is most likely no warp drive possible and even planets in the Goldilocks zone will have unpleasant viral and cellular agents). Biological alien creatures will be too adapted to their own planets.

As to extinction, we will only not go extinct if our robot masters decide to keep some of us around.  If they decide to populate new planets with human life then they could make the journey and humans would thrive (but only because the synthetic agents wanted this).

If a flying saucer ever lands, the chances are 99.99% that what steps out will be a synthetic intelligent entity.   It's just too hard for biological entities (adapted to their planet) to make the long voyages required.

Peter Gacs: I give it a probability near 1%.  Humans may become irrelevant in the sense of losing their role of being at the forefront of the progress of "self-knowledge of the universe" (whatever this means).  But irrelevance will also mean that it will not be important to eradicate them completely.  On the other hand, there are just too many, too diverse imaginable scenarios for their coexistence with machines that are smarter than they are, so I don't dare to predict any details.  Of course, species do die out daily even without our intent to extinguish them, but I assume that at least some humans would find ways to survive for some more centuries to come.

Eray Ozkural: Assuming that we are talking about intelligent agents, which are strictly unnecessary for working on scientific problems which is your main concern, I think first that it is not possible to build something that is provably non-dangerous, unless you can encode a rule of non-interference into its behavior. Otherwise, an interfering AI can basically do anything, and since it is much smarter than us, it can create actual problems that we had no way of anticipating or solving. I have thought at length on this question, and considered some possible AI objectives in a blog essay:

http://www.examachine.net/blog/?p=72

I think that it does depend on the objectives. In particular, selfish/expansionist AI objectives are very dangerous. They might almost certainly result in interference with our vital resources. I cannot give a probability, because it is a can of worms, but let me try to summarize. For instance, the objective to maximize its knowledge about the world, a similar version of which was considered by Laurent Orseau in a reinforcement learning setting, and previously by a student of Solomonoff. Well, it's an intuitive idea that a scientist tries to learn as much as possible about the world. What if we built an intelligent agent that did that? If it's successful, it would have to increase its computation and physical capacity to such an extent that it might expand rapidly, first assimilating the solar system and then expand at our galactic neighborhood to be able to pursue its unsatisfiable urge to learn. Similar scenarios might happen in any kind of intelligent agent with selfish objectives (i.e., optimize some aspect of itself). Those might be recognized as Omohundro drives, but the objectives themselves are the main problem mostly.

This is a problem when you are stuck in this reinforcement learning mentality, thinking in terms of rewards and punishment. The utility function that you will define will tend to be centered around the AI itself rather than humanity, and things have a good chance of going very wrong. This is mostly regardless of what kind of selfishness is pursued, be it knowledge, intelligence, power, control, satisfaction of pseudo pleasure, etc. In the end, the problem is with the relentless pursuit of a singular, general objective that seeks to benefit only the self. And this cannot be mitigated by any amount of obstruction rules (like Robot Laws or any other kind of laws). The motivation is what matters, and even when you are not pursuing silly motivations like stamp collection, there is a lot of danger involved, not due to our neglect of human values, which are mostly irrelevant at the level which such an intelligent agent would operate, but our design of its motivations.

However, even if benevolent looking objectives were adopted, it is not altogether clear, what sorts of crazy schemes an AI would come up with. In fact, we could not predict the plans of an intelligent agent smarter than the entire humanity. Therefore, it's a gamble at best, and even if we made a life-loving, information-loving, selfless, autonomous AI as I suggested, it might still do a lot of things that many people would disagree with. And although such an AI might not extinguish our species, it might decide, for instance, that it would be best to scan and archive our species for using later. That is, there is no reason to expect that an intelligent agent that is superior to us in every respect should abide by our will.

One might try to imagine many solutions to make such intelligent agents "fool-proof" and "fail-safe", but I suspect that for the first, human foolishness has unbounded inventiveness, and for the second, no amount of security methods that we design would make a mind that is smarter than the entire humanity "safe", as we have no way of anticipating every situation that would be created by its massive intelligence, and the amount of chaotic change that it would bring. It would simply go out of control, and we would be at the mercy of its evolved personality. I said personality on purpose, because personality seems to be a result of initial motivations, a priori knowledge, and its life experience. Since its life experience and intelligence will overshadow any initial programming, we cannot really foresee its future personality. All in all, I think it is great for thinking about, but it does not look like a practical engineering solution. That's why I simply advise against building fully autonomous intelligent agents. I sometimes say, play God, and you will fail. I tend to think there is a Frankenstein Complex, it is as if there is an incredible urge in many people to create an independent artificial person.

On the other hand, I can imagine how I could build semi-autonomous agents that might be useful for many special tasks, avoiding interference with humans as much as possible, with practical ways to test for their compliance with law and customs. However, personally speaking, I cannot imagine a single reason why I would want to  create an artificial person that is superior to me in every respect. Unless of course, I have elected to bow down to a superior species.

Laurent Orseau: It depends if we consider that we will simply leave safety issues aside before creating an AGI, thinking that all will go well, or if we take into account that we will actually do some research on that.
If an human-level AGI was built today, then we probably wouldn't be ready and the risks due to excitement to get something out of it might be high ("hey look, it can drive the tank, how cool is that?!").

But if we build one and can show to the world a simple proof of concept that we do have (sub-human level) AGI and that will grow to human-level and most researchers acknowledge it, I presume we will start to think hard about the consequences.

Then all depends on how much unfriendly it is.
Humanity is intelligent enough to care for its own life, and try to avoid high risks (most of the time), unless there is some really huge benefit (like supremacy).

Also, if an AGI wants to kill all humans, humanity would not just wait for it, doing nothing.
This might be dangerous for the AI itself too (with EMPs for example). And an AGI also wants to avoid high risks unless there is a huge benefit. If some compromise is possible, it should be better.

If we can build an AGI that is quite friendly (i.e. has "good" goals and wants to cooperate with humans without pressing them too much, or at least has no incentive to kill humans) but may become nasty only if its life is at stake, then I don't think we need to worry *too* much: just be friendly with it as you would be with an ally, and its safety will be paired with your own safety.

So I think the risks of human extinction will be pretty low, as long as we take them into account seriously.

Richard Loosemore: The question is loaded, and I reject the premises. It assumes that someone can build an AI that is both generally intelligent (enough to be able to improve itself) whilst also having a design whose motivation is impossible to prove. That is a false assumption. People who try to build AI systems with the kind of design whose motivation is unstable will actually not succeed in building anything that has enough general intelligence to become a danger.

Monica Anderson: 0.00%. All intelligences must be fallible in order to deal with a complex and illogical world (with only incomplete information available) on a best effort basis. And if an AI is fallible, then we can unplug it... sooner or later, even if it is "designed to be unstoppable". Ten people armed with pitchforks, and armed also with ten copies of last year's best AI can always unplug the latest model AI.

The Interview (Old Questions)

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

Leo Pape: For me, roughly human-level machine intelligence is an embodied machine. Given the current difficulties of making such machines I expect it will last at least several hundred years before human-level intelligence can be reached. Making better machines is not a question of superintelligence, but of long and hard work. Try getting some responses to your questionnaire from roboticists.

Donald Loveland: Experts usually are correct in their predictions but terrible in their timing predictions. They usually see things as coming earlier than the event actually occurs as they fail to see the obstacles. Also, it is unclear what you mean as human-level intelligence. The Turing test will be passed in its simplest form perhaps in 20 years. Full functional replacements for humans will likely take over 100 years (50% likelihood). 200 years (90% likelihood).

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Explanatory remark to Q2:

P(human extinction | badly done AI) = ?

(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)

Leo Pape: Human beings are already using all sorts of artificial intelligence in their (war)machines, so there it is not impossible that our machines will be helpful in human extinction.

Donald Loveland: Ultimately 95% (and not just by bad AI, but just by generalized evolution). In other words, in this sense all AI is badly done AI for I think it is a natural sequence that AI leads to superior artificial minds that leads to eventual evolution, or replacement (depending on the speed of the transformation), of humans to artificial life.

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?

Leo Pape: I don’t know what "massive superhuman intelligence" is, what it is for, and if it existed how to measure it.

Donald Loveland: I am not sure of the question, or maybe only that I do not understand an answer. Let me comment anyway. I have always felt it likely that the first superhuman intelligence would be a simulation of the human mind; e.g., by advanced neural-net-like structures. I have never thought seriously about learning time, but I guess the first success would be after some years of processing. I am not sure of what you mean by ``massive''. Such a mind as above coupled to good retrieval algorithms with extensive databases such as those being developed now could appear to have massive superhuman intelligence.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Explanatory remark to Q4:

How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

Leo Pape: Proof is for mathematics, not for actual machines. Even for the simplest machines we have nowadays we cannot proof any aspect of their operation. If this were possible, airplane travel would be a lot safer.

Donald Loveland: It is important to try. I do not think it can be done. I feel that humans are safe from AI takeover for this century. Maybe not from other calamities, however.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Explanatory remark to Q5:

What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

Leo Pape: No idea how to compare these risks.

Donald Loveland: Ouch. Now you want me to amplify my casual remark above. I guess that I can only say that I hope we are lucky enough for the human race to survive long enough to evolve into, or be taken over by, another type of intelligence.

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

Leo Pape: People think that current AI is much more capable than it is in reality, and therefore they often overestimate the risks. This is partly due to the movies and due to scientists overselling their work in scientific papers and in the media. So I think the risk is highly overestimated.

Donald Loveland: The current level of awareness of the AI risks is low. The risk that I most focus on now is the economic repercussions of advancing AI. Together with outsourcing, the advancing automation of the workplace, now dominated by AI advances,  is leading to increasing unemployment. This progression will not be monotonic, but each recession will result in more permanently unemployed and weaker recoveries. At some point our economic philosophy could change radically in the U.S., an event very similar to the great depression. We may not recover, in the sense of returning to the same economic structure. I think (hope) that democracy will survive.

Q7: Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?

Leo Pape: I would be impressed if a team of soccer playing robots could win a match against professional human players. Of course, the real challenge is finding human players that are willing to play against machines (imagine being tackled by a metal robot).

Donald Loveland: A "pure" learning program that won at Jeopardy ???

Q8: Are you familiar with formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter's algorithm M, AIXI, or Gödel machines?

Donald Loveland: I have some familiarity with Solomonoff inductive inference but not Hutter's algorithm. I have been retired for 10 years so didn't know of Hutter until this email. Looks like something interesting to pursue.

Q&A with Abram Demski on risks from AI

22 XiXiDu 17 January 2012 09:43AM

[Click here to see a list of all interviews]

Abram Demski is a computer science Ph.D student at the University of Southern California who has previously studied cognitive science at Central Michigan University. He is an artificial intelligence enthusiast looking for the logic of thought. He is interested in AGI in general and universal theories of intelligence in particular, but also probabilistic reasoning, logic, and the combination of the two ("relational methods"). Also, utility-theoretic reasoning.

I interviewed Abram Demski due to feedback from lesswrong. cousin_it, top contributer and research associate of the Singularity institute, wrote the following:

I'm afraid of Abram Demski who wrote brilliant comments on LW and still got paid to help design a self-improving AGI (Genifer).

Enough already, here goes....

The Interview:

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

Abram Demski:

10%: 5 years (2017).
50%: 15 years (2027).
90%: 50 years (2062).

Of course, just numbers is not very informative. The year numbers I gave are unstable under reflection, at a factor of about 2 (meaning I have doubled and halved these estimates in the past minutes while considering it). More relevant is the variance; I think the year of development is fundamentally hard to predict, so that it's rational to give a significant probability mass to within 10 years, but also to it taking another 50 years or more. However, the largest bulk of my probability mass would be roughly between 2020 and 2030, since (1) the computing hardware to simulate the human brain would become widely available and (2) I believe less than that will be sufficient, but the software may lag behind the hardware potential by 5 to 10 years. (I would estimate more lag, except that it looks like we are making good progress right now.)

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Explanatory remark to Q2:

P(human extinction | badly done AI) = ?

(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)

Abram Demski: This is somewhat difficult. We could say that AIs matching that description have already been created (with few negative consequences). I presume that "roughly human-level" is also intended, though.

If the human-level AGI

0) is autonomous (has, or forms, long-term goals)
1) is not socialized
2) figures out how to access spare computing power on the internet
3) has a goal which is very bad for humans (ie, implies extinction)
4) is alone (has no similarly-capable peers)

then the probability of human extinction is quite high, though not 1. The probability of #0 is somewhat low; #1 is somewhat low; #2 is fairly high; #3 is difficult to estimate; #4 is somewhat low.

#1 is important because a self-modifying system will tend to respond to negative reinforcement concerning sociopathic behaviors resulting from #3-- though, it must be admitted, this will depend on how deeply the ability to self-modify runs. Not all architectures will be capable of effectively modifying their goals in response to social pressures. (In fact, rigid goal-structure under self-modification will usually be seen as an important design-point.)

#3 depends a great deal on just how smart the agent is. Given an agent of merely human capability, human extinction would be very improbable even with an agent that was given the explicit goal of destroying humans. Given an agent of somewhat greater intelligence, the risk would be there, but it's not so clear what range of goals would be bad for humans (many goals could be accomplished through cooperation). For a vastly more intelligent agent, predicting behavior is naturally a bit more difficult, but cooperation with humans would not be as necessary for survival. So, that is why #2 becomes very important: an agent that is human-level when run on the computing power of a single machine (or small network) could be much more intelligent with access to even a small fraction of the world's computing power.

#4 is a common presumption in singularity stories, because there has to be a first super-human AI at some point. However, the nature of software is such that once the fundamental innovation is made, creating and deploying many is easy. Furthermore, a human-like system may have a human-like training time (to become adult-level that is), in which case it may have many peers (which gets back to #1). In case #4 is *not* true, then condition #3 must be rewritten to "most such systems have goals which are bad for humans".

It's very difficult to give an actual probability estimate for this question because of the way "badly done AI" pushes around the probability. (By definition, there should be some negative consequences, or it wasn't done badly enough...) However, I'll naively multiply the factors I've given, with some very rough numbers:

P(#0)P(#1)P(#2)P(#3)P(#4)
= .1 * .1 * .9 * .5 * .1
= .00045

I described a fairly narrow scenario, so we might expect significant probability mass to come from other possibilities. However, I think it's the most plausible. So, keeping in mind that it's very rough, let's say .001.

I note that this is significantly lower than estimates I've made before, despite trying harder at that time to refute the hypothesis.

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?

Abram Demski: Very near zero, very near zero, and very near zero. My feeling is that intelligence is a combination of processing power and knowledge. In this case, knowledge will keep pouring in, but processing power will become a limiting factor. Self-modification does not help this. So, such a system might become superhuman within 5 years, but not massively.

If the system does copy itself or otherwise gain more processing power, then I assign much higher probability; 1% within hours, 5% within days, 90% within 5 years.

Note that there is a very important ambiguity in the term "human-level", though. It could mean child-level or adult-level. (IE, a human-level system may take 20 years to train to the adult level.) The above assumes you mean "adult level". If not, add 20 years.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Abram Demski: "Provably non-dangerous" may not be the best way of thinking about the problem. Overall, the goal is to reduce risk. Proof may not be possible or may not be the most effective route.

So: is it important to solve the problem of safety before trying to solve the problem of intelligence?

I don't think this is possible. Designs for safe systems have to be designs for systems, so they must be informed by solutions to the intelligence problem.

It would also be undesirable to stall progress while considering the consequences. Serious risks are associated with many areas of research, but it typically seems better to mitigate those risks while moving forward rather than beforehand.

That said, it seems like a good idea to put some thought into safety & friendliness while we are solving the general intelligence problem.

Q4-sub: How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

Abram Demski: I am a bias authority for this question, but in general, increased funding to AI would be good news to me. My opinion is that the world has a lot of problems which we are slowly solving, but which could be addressed much more effectively if we had more intelligence with which to attack the problem. AI research is beginning to bear fruits in this way (becoming a profitable industry).

So, if I had my say, I would re-assign perhaps half the budget currently assigned to the military to AI research. (The other half would go to NASA and particle physicists...)

What amount of the AI budget should be concentrated on safety? Moderately more than at present. Almost no one is working on safety right now.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Abram Demski: I would rank nanotech as fairly low on my list of concerns, because cells are fairly close to optimal replicators in the present environment. (IE, I don't buy the grey-goo stories: the worst that I find plausible is a nanobot plague, and normal biological weapons would be easier to make.)

Anyway, AI would be lower on my list than global warming.

Q5-sub: What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

Abram Demski: Another world war with the present technology & weapon stockpiles (or greater).

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

Abram Demski: The general population seems to be highly aware of the risks of AI, with very little awareness of the benefits.

Within the research community, the situation was completely opposite until recently. I would say present awareness levels in the research community is roughly optimal...

Q7: Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?

Abram Demski: No. Predictions about AI research have historically been mostly wrong, so it would be incorrect to make such predictions.

Q8: Are you familiar with formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter's algorithm M, AIXI, or Gödel machines?

Abram Demski: Yes.

Addendum

Abram Demski: I have been, and am, generally conflicted about these issues. I first encountered the writings of Eliezer three or four years ago. I was already a Bayesian at the time, and I was working with great conviction on the problem of finding the "one true logic" which could serve as a foundation of reasoning. Central to this was finding the correct formal theory of truth (a problem which is hard thanks to the Liar paradox). Reading Eliezer's material, it was clear that he would be interested in the same things, but wasn't writing a great deal about them in public.

I sent him an email about it. (I had the good habit of emailing random researchers, a habit I recommend to anyone.) His response was that he needed to speak to me on the phone before collaborating with me, since much more about a person is conveyed by audio. So, we set up a phone call.

I tried to discuss logic on the phone, and was successful for a few minutes, but Eliezer's purpose was to deliver the argument for existential risk from AI as a set-up for the central question which would determine whether he would be willing to work with me: If I found the correct logic, would I publish? I answered yes, which meant that we could not work together. The risk for him was too high.

Was I rational in my response? What reason should I have to publish, that would outweigh the risk of someone taking the research and misusing it? (Eliezer made the comment that with the correct logic in hand, it might take a year to implement a seed AI capable of bootstrapping itself to superhuman intelligence.) My perception of my own thought process is much closer to "clinging to normality" than one of "carefully evaluating the correct position". Shortly after that, trying to resolve my inner conflict (prove to myself that I was being rational) I wrote this:

http://dragonlogic-ai.blogspot.com/2009/04/some-numbers-continues-risk-estimate.html

The numbers there just barely indicate that I should keep working on AI (and I give a cut-off date, beyond which it would be too dangerous). Despite trying hard to prove that AI was not so risky, I was experiencing an anchoring bias. AI being terribly risky was the start position from which estimates should be improved.

Still, although I attempted to be fair in my new probability estimates for this interview, it must be admitted that my argument took the form of listing factors which might plausibly reduce the risk and multiplying them together so that the risk gets smaller and smaller. Does this pattern reflect the real physics of the situation, or does it still reflect anchor-and-reduce type reasoning? Have I encountered more factors to reduce the probability because that's been what I've looked for?

My central claims are:

  • In order for destruction of humanity to be a good idea, the AI would have to be so powerful that it could simply brush humanity aside, or have a very malevolent goal, or both.
  • In order to have massively superhuman intelligence, massively more processing power is needed (than is needed to achieve human-level intelligence).
  • It seems unlikely that a malevolent system would emerge in an environment empty of potential rivals which may be more pro-human, which further increases the processing power requirement (because the malevolent system should be much more powerful than rivals in order for cooperation to be an unimportant option), while making massive acquisition of such processing power (taking over the internet single-handedly) less plausible.
  • In any case, these negative singularity scenarios tend to assume an autonomous AI system (one with persistent goals). It is more likely that the industry will focus on creating non-autonomous systems in the near-term, and it seems like these would have to become autonomous to be dangerous in that way. (Autonomous systems will be created more for entertainment & eventually companionship, in which case much more attention will be paid to friendly goal systems.)

Some danger comes from originally non-autonomous systems which become autonomous (ie, whose capabilities are expanded to general planning to maximize some industrially useful utility function such as production quality, cash flow, ad clicks, etc). These are more likely to have strange values not good for us. The hope here lies in the possibility that these would be so numerous that cooperation with society & other AIs would be the only option. A scenario where the first AI capable of true recursive self improvement became an industrial AI and rose to power before its makers could re-sell it to many different jobs seems unlikely (because the loop between research and industry is not usually like that). More likely, by the time the first human-level system is five years old (old enough to start thinking about world domination), different versions of it with different goals and experiences will be in many places having many experiences, cooperating with humankind more than each other.

But, anyway, all of this is an argument that the probability is low, not that it would be impossible or that the consequences aren't hugely undesirable. That's why I said the level of awareness in the AI community is good and that research into safe AI could use a bit more funding.

Q&A with experts on risks from AI #3

13 XiXiDu 12 January 2012 10:45AM

[Click here to see a list of all interviews]

I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI.

Dr. Pei Wang is trying to build general-purpose AI systems, compare them with human intelligence, analyze their theoretical assumptions, and evaluate their potential and limitation. [Curriculum Vitae] [Pei Wang on the Path to Artificial General Intelligence]

Dr. J. Storrs Hall is an independent scientist and author. His most recent book is Beyond AI: Creating the Conscience of the Machine, published by Prometheus Books. It is about the (possibly) imminent development of strong AI, and the desirability, if and when that happens, that such AIs be equipped with a moral sense and conscience. This is an outgrowth of his essay Ethics for Machines. [Homepage]

Professor Paul Cohen is the director of the School of Information: Science, Technology, and Arts at the University of Arizona. His research is in artificial intelligence. He wants to model human cognitive development in silico, with robots or softbots in game environments as the "babies" they're trying to raise up. he is particularly interested in the sensorimotor foundations of human language. Several of his projects in the last decade have developed algorithms for sensor-to-symbol kinds of processing in service of learning the meanings of words, most recently, verbs. He also works in what they call Education Informatics, which includes intelligent tutoring systems, data mining and statistical modeling of students' mastery and engagement, assessment technologies, ontologies for representing student data and standards for content, architectures for content delivery, and so on. [Homepage]

The Interview:

Q1: Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans at science, mathematics, engineering and programming?

Pei Wang: My estimations are, very roughly, 2020/2030/2050, respectively.

Here by "roughly as good as humans" I mean the AI will follow roughly the same principles as human in information processing, though it does not mean that the system will have the same behavior or capability as human, due to the difference in body, experience, motivation, etc.

J. Storrs Hall: 2020 / 2030 / 2040

Paul Cohen: I wish the answer were simple.  As early as the 1970s, AI programs were making modest scientific discoveries and discovering (or more often, rediscovering) bits of mathematics.  Computer-based proof checkers are apparently common in math, though I don't know anything about them.  If you are asking when machines will function as complete, autonomous scientists (or anything else) I'd say there's little reason to think that that's what we want.  For another few decades we will be developing assistants, amplifiers, and parts of the scientific/creative process.  There are communities who strive for complete and autonomous automated scientists, but last time I looked, a couple of years back, it was "look ma, no hands" demonstrations with little of interest under the hood. On the other hand, joint machine-human efforts, especially those that involve citizen scientists (e.g., Galaxy Zoo, Foldit) are apt to be increasingly productive.

Q2: Once we build AI that is roughly as good as humans at science, mathematics, engineering and programming, how much more difficult will it be for humans and/or AIs to build an AI which is substantially better at those activities than humans?

Pei Wang: After that, AI can become more powerful (in hardware), more knowledgeable, and therefore more capable in problem solving, than human beings. However, there is no evidence to believe that it can be "substantially better" in the principles defining intelligence.

J. Storrs Hall: Difficult in what sense?  Make 20 000 copies of your AI and organize them as Google or Apple. The difficulty is economic, not technical.

Paul Cohen: It isn't hard to do better than humans. The earliest expert systems outperformed most humans. You can't beat a machine at chess. etc.  Google is developing cars that I think will probably drive better than humans. The Google search engine does what no human can. 

Q3: Do you ever expect artificial intelligence to overwhelmingly outperform humans at typical academic research, in the way that they may soon overwhelmingly outperform humans at trivia contests, or do you expect that humans will always play an important role in scientific progress?

Pei Wang: Even when AI follows the same principles as humans, and has more computational power and other resources than humans, they won't "overwhelmingly outperform humans" in all activities, due to the difference in hardware, experience, and motivations. There will always be some tasks that humans do better, and others that machines do better.

J. Storrs Hall: A large part of academic research is entertainment and status fights, and it doesn't really matter whether machines are good at that or not.  A large part of scientific research and technical development is experimentation and data gathering, and these are mostly resource-limited rather than smarts-limited. increasing AI intelligence doesn't address the bottleneck.

Paul Cohen: One fundamental observation from sixty years of AI is that generality is hard, specialization is easy.  This is one reason that Watson is a greater accomplishment than Deep Blue.  Scientists specialize (although, arguably, the best scientists are not ultra-specialists but maintain a broad-enough perspective to see connections that lead to new work).  So a narrow area of science is easier than the common sense that a home-help robot will need.   I think it's very likely that in some areas of science, machines will do much of the creative work and also the drudge work.

Q4: What probability do you assign to the possibility of an AI with initially (professional) human-level competence at general reasoning (including science, mathematics, engineering and programming) to self-modify its way up to vastly superhuman capabilities within a matter of hours/days/< 5 years?

Pei Wang: Though there are speculations for such a "self-modifying to superhuman" scenario, all of them contains various wrong or unsupported assumptions. I haven't been convinced for such a possibility at all. It is possible for AI systems to become more and more capable, but I don't think they will become completely uncontrollable or incomprehensible.

J. Storrs Hall: This depends entirely on when it starts, i.e. what is the current marginal cost of computation along the Moore's Law curve. A reductio ad adsurdum: in 1970, when all the computers in the world might possibly have sufficed to run one human-equivalent program, the amount of work it would have had to do to improve to superhuman would be way out of its grasp. In 2050, it will probably be trivial, since computation will be extremely cheap and the necessary algorithms and knowledge bases will likely be available as open source.

Paul Cohen: The first step is the hardest:  "human level competence at general reasoning" is our greatest challenge.  I am quite sure that anything that could, say, read and understand what it reads would in a matter of days, weeks or months become vastly more generative than humans.  But the first step is still well beyond our grasp.

Q5: How important is it to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to build AI that is good enough at general reasoning (including science, mathematics, engineering and programming) to undergo radical self-modification?

Pei Wang: I think the idea "to make superhuman AI provably friendly" is similar to the idea "to make airplane provably safe" and "to make baby provably ethical" --- though the motivation is respectful, the goal cannot be accurately defined, and the approximate definitions cannot be reached.

What if the Wright brothers were asked "to figure out how to make airplane provably safe before attempting to build it", or all parents are asked "to figure out how to make children provably ethical before attempting to have them"?

Since an AI system is adaptive (according to my opinion, as well as many others'), its behaviors won't be fully determined by its initial state or design (nature), but strongly influenced by its experience (nurture). You cannot make a friendly AI (whatever it means), but have to educate an AI to become friendly. Even in that case, it cannot be "provably friendly" --- only mathematical conclusions can be proved, and empirical predictions are always fallible.

J. Storrs Hall: This is approximately like saying we need to require a proof, based on someone's DNA sequence, that they can never commit a sin, and that we must not allow any babies to be born until they can offer such a proof.

Paul Cohen:  Same answer as above. Today we can build ultra specialist assistants (and so maintain control and make the ethical decisions ourselves) and we can't go further until we solve the problems of general intelligence -- vision, language understanding, reading, reasoning...

Q6: What probability do you assign to the possibility of human extinction as a result of AI capable of self-modification (that is not provably non-dangerous, if that is even possible)? P(human extinction by AI | AI capable of self-modification and not provably non-dangerous is created)

Pei Wang: I don't think it makes much sense to talk about "probability" here, except to drop all of its mathematical meaning.

Which discovery is "provably non-dangerous"?  Physics, chemistry, and biology are all responsible for known ways to human extinction. Should we pause all these explorations until they are "provably safe"? How about the use of fire? Would the human species do better without using this "provably dangerous" technique?

AI systems, like all major scientific and technical results, can lead to human extinction, but it is not the reason to stop or pause this research. Otherwise we cannot do anything, since every non-trivial action has unanticipated consequences. Though it is important to be aware of the potential danger of AI, we probably have no real alternative but to take this opportunity and challenge, and making our best decisions according to their predicted consequences.

J. Storrs Hall:  This is unlikely but not inconceivable.  If it happens, however, it will be because the AI was part of a doomsday device probably built by some military for "mutual assured destruction", and some other military tried to call their bluff. The best defense against this is for the rest of the world to be as smart as possible as fast as possible.

To sum up, AIs can and should be vetted with standard and well-understood quality assurance and testing techniques, but defining "friendliness to the human race", much less proving it, is a pipe dream.

Paul Cohen: From where I sit today, near zero.  Besides, the danger is likely to be mostly on the human side: Irrespective of what machines can or cannot do, we will continue to be lazy, self-righteous, jingoistic, squanderers of our tiny little planet. It seems to me much more likely that we destroy will our universities and research base and devote ourselves to wars over what little remains of our water and land.  If the current anti-intellectual rhetoric continues, if we continue to reject science for ignorance and God, then we will first destroy the research base that can produce intelligent machines and then destroy the planet.  So I wouldn't worry too much about Dr. Evil and her Annihilating AI.  We have more pressing matters to worry about.

William Uther

Dr. William Uther [Homepage] answered two sets of old questions and also made some additional comments.

William Uther: I can only answer for myself, not for my employer or anyone else (or even my future self).

I have a few comments before I start:

  • You ask a lot about 'human level AGI'.  I do not think this term is well defined.  It assumes that 'intelligence' is a one-dimensional quantity.  It isn't.  We already have AI systems that play chess better than the best humans, and mine data (one definition of 'learn') better than humans.  Robots can currently drive cars roughly as well as humans can.  We don't yet have a robot than can clean up a child's messy bedroom.  Of course, we don't have children that can do that either. :)
  • Intelligence is different from motivation.  Each is different from consciousness.  You seem to be envisioning a robot as some sort of super-human, self-motivated, conscious device.  I don't know any AI researchers working towards that goal.  (There may well be some, but I don't know them.)  As such the problems we're likely to have with AI are less 'Terminator' and more 'Sorcerer's apprentice' (see http://en.wikipedia.org/wiki/The_Sorcerer's_Apprentice ).  These types of problems are less worrying as, in general, the AI isn't trying to actively hurt humans.
  • As you bring up in one of you later questions, I think there are far more pressing worries at the moment than AI run amok.

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

William Uther: As I said above, I don't think this question is well specified.  It assumes that 'intelligence' is a one-dimensional quantity.  It isn't.  We already have AI systems that play chess better than the best humans, and mine data (one definition of learn) better than humans.  Robots can currently drive cars roughly as well as humans can.  We don't yet have a robot than can clean up a child's messy bedroom.  Of course, we don't have children that can do that either. :)

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Explanatory remark to Q2:

P(human extinction | badly done AI) = ?

(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)

William Uther: Again, I don't think your question is well specified.  Most AI researchers are working on AI as a tool: given a task, the AI tries to figure out how to do it.  They're working on artificial intelligence, not artificial self-motivation.  I don't know that we could even measure something like 'artificial consciousness'.

All tools increase the power of those that use them.  But where does the blame lie if something goes wrong with the tool?  In the terms of the US gun debate: Do guns kill people?  Do people kill people?  Do gun manufacturers kill people?  Do kitchen knife manufacturers kill people?

Personally, I don't think 'Terminator' style machines run amok is a very likely scenario.  Hrm - I should be clearer here.  I believe that there are already AI systems that have had malfunctions and killed people (see http://www.wired.com/dangerroom/2007/10/robot-cannon-ki/ ).  I also believe that when fire was first discovered there was probably some early caveman that started a forest fire and got himself roasted.  He could even have roasted most of his village.  I do not believe that mankind will build AI systems that will systematically seek out and deliberately destroy all humans (e.g. 'Skynet'), and I further believe that if someone started a system like this it would be destroyed by everyone else quite quickly.

It isn't hard to build in an 'off' switch.  In most cases that is a very simple solution to 'Skynet' style problems.

I think there are much more worrying developments in the biological sciences.  See http://www.nytimes.com/2012/01/08/opinion/sunday/an-engineered-doomsday.html

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?

William Uther: Again, your question is poorly specified.  What do you mean by 'human level AGI'?  Trying to tease this apart, do you mean a robotic system that if trained up for 20 years like a human would end up as smart as a human 20-year-old? Are you referring to that system before the 20 years learning, or after?

In general, if the system has 'human level' AGI, then surely it will behave the same way as a human.  In which case none of your scenarios are likely - I've had an internet connection for years and I'm not super-human yet.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Explanatory remark to Q4:

How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

William Uther: I think this is a worthwhile goal for a small number of researchers to think about, but I don't think we need many.  I think we are far enough away from 'super-intelligences' that it isn't urgent.  In particular, I don't think that having 'machines smarter than humans' is some sort of magical tipping point.  AI is HARD.  Having machines that are smarter than humans means they'll make progress faster than humans would.  It doesn't mean they'll make progress massively faster than humans would in the short term.

I also think there are ethical issues worth considering before we have AGI.  See http://m.theatlantic.com/technology/print/2011/12/drone-ethics-briefing-what-a-leading-robot-expert-told-the-cia/250060/

Note that none of those ethical issues assume some sort of super-intelligence.  In the same that ethics in humans doesn't assume super-intelligence.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Explanatory remark to Q5:

What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

William Uther: I have a few worries.  From the top of my head:

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

William Uther: I think most people aren't worried about AI risks.  I don't think they should be.  I don't see a problem here.

Q7: Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?

William Uther: I still don't know what you mean by 'human level intelligence'.  I expect artificial intelligence to be quite different to human intelligence.  AI is already common in many businesses - if you have a bank loan then the decision about whether to lend to you was probably taken by a machine learnt system.

Q1a: Assuming beneficially political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of machine intelligence with roughly human-level efficient cross-domain optimization competence?

Q1b: Once our understanding of AI and our hardware capabilities are sufficiently sophisticated to build an AI which is as good as humans at engineering or programming, how much more difficult will it be to build an AI which is substantially better than humans at mathematics or engineering?

William Uther: There is a whole field of 'automatic programming'.  The main difficulties in that field were in specifying what you wanted programmed.  Once you'd done that the computers were quite effective at making it.  (I'm not sure people have tried to make computers design complex algorithms and data structures yet.)

Q2a: Do you ever expect automated systems to overwhelmingly outperform humans at typical academic research, in the way that they may soon overwhelmingly outperform humans at trivia contests, or do you expect that humans will always play an important role in scientific progress?

William Uther: I think asking about 'automated science' is a much clearer question than asking about 'Human level AGI'.  At the moment there is already huge amounts of automation in science (from Peter Cheeseman's early work with AutoClass to the biological 'experiments on a chip' that allow a large number of parallel tests to be run).  What is happening is similar to automation in other areas - the simpler tasks (both intellectual and physical) are being automated away and the humans are working at higher levels of abstraction.  There will always be *a* role for humans in scientific research (in much the same way that there is currently a role for program managers in current research - they decide at a high level what research should be done after understanding as much of it as they choose).

Q2b: To what extent does human engineering and mathematical ability rely on many varied aspects of human cognition, such as social interaction and embodiment? For example, would an AI with human-level skill at mathematics and programming be able to design a new AI with sophisticated social skills, or does that require an AI which already possesses sophisticated social skills?

William Uther: Social skills require understanding humans.  We have no abstract mathematical model of humans as yet to load into a machine, and so the only way you can learn to understand humans is by experimenting on them... er, I mean, interacting with them. :)  That takes time, and humans who are willing to interact with you.

Once you have the model, coming up with optimal plans for interacting with it, i.e. social skills, can happen offline.  It is building the model of humans that is the bottleneck for an infinitely powerful machine.

I guess you cold parallelise it by interacting with each human on the planet simultaneously.  That would gather a large amount of data quite quickly, but be tricky to organise.  And certain parts of learning about a system cannot be parallelised.

Q2c: What probability do you assign to the possibility of an AI with initially (professional) human-level competence at mathematics and programming to self-modify its way up to massive superhuman efficient cross-domain optimization competence within a matter of hours/days/< 5 years?

William Uther: One possible outcome is that we find out that humans are close to optimal problem solvers given the resources they allocate to the problem.  In which case, 'massive superhuman cross-domain optimisation' may simply not be possible.

Humans are only an existence proof for human level intelligence.

Q7: How much have you read about the formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter's algorithm M, AIXI, or Gödel machines?

William Uther: I know of all of those.  I know some of the AIXI approximations quite well.  The lesson I draw from all of those is that AI is HARD.  In fact, the real question isn't how do you perform optimally, but how do you perform well enough given the resources you have at hand.  Humans are a long way from optimal, but they do quite well given the resources they have.

I'd like to make some other points as well:

  • When trying to define 'Human level intelligence' it is often useful to consider how many humans meet your standard.  If the answer is 'not many' then you don't have a very good measure of human level intelligence.  Does Michael Jordan (the basketball player) have human level intelligence?  Does Stephen Hawking?  Does George Bush?
  • People who are worried about the singularity often have two classes of concerns.  First there is the class of people who worry about robots taking over and just leaving humans behind.  I think that is highly unlikely.  I think it much more likely that humans and machines will interact and progress together.  Once I have my brain plugged in to an advanced computer there will be no AI that can out-think me.  Computers already allow us to 'think' in ways that we couldn't have dreamt of 50 years ago.

This brings up the second class of issues that people have.  Once we are connected to machines, will we still be truly human.  I have no idea what people who worry about this mean by 'truly human'.  Is a human with a prosthetic limb truly human?  How about a human driving a car?  Is a human who wears classes or a hearing aid truly human?  If these prosthetics make you non-human, then we're already past the point where they should be concerned - and they're not.  If these prosthetics leave you human, then why would a piece of glass that allows me to see clearly be ok, and a computer that allows me to think clearly not be ok?  Asimov investigated ideas similar to this, but from a slightly different point of view, with his story 'The Bicentennial Man'.

The real questions are ones of ethics.  As people become more powerful, what are the ethical ways of using that power?  I have no great wisdom to share there, unfortunately.

Some more thoughts...

Does a research lab (with, say, 50 researchers) have "above human level intelligence"?  If not, then it isn't clear to me that AI will ever have significantly "above human level intelligence" (and see below for why AI is still worthwhile).  If so, then why haven't we had a 'research lab singularity' yet?  Surely research labs are smarter than humans and so they can work on making still smarter research labs, until a magical point is passed and research labs have runaway intelligence.  (That's a socratic question designed to get you to think about possible answers yourself.  Maybe we are in the middle of a research lab singularity.)

As for why study of AI might still be useful even if we never get above human level intelligence: there is the same Dirty, Dull, Dangerous argument that has been used many times.  To that I'd add a point I made in a previous email: intelligence is different to motivation.  If you get yourself another human you get both - they're intelligent, but they also have their own goals and you have to spend time convincing them to work towards your goals.  If you get an AI, then even if it isn't more intelligent than a human at least all that intelligence is working towards your goals without argument.  It's similar to the 'Dull' justification, but with a slightly different spin.

Alan Bundy

Professor Alan Bundy [homepage] did not answer my questions directly.

Alan Bundy: Whenever I see questions like this I want to start by questioning the implicit assumptions behind them.

  • I don't think the concept of "human-level machine intelligence" is well formed. AI is defining a huge space of different kinds of intelligence. Most of the points in this space are unlike anything either human or animal, but are new kinds of intelligence. Most of them are very specialised to particular areas of expertise. As an extreme example, consider the best chess playing programs. They are more intelligent than any human at playing chess, but can do nothing else, e.g., pick up and move the pieces. There's a popular myth that intelligence is on a linear scale, like IQ, and AI is progressing along it. If so, where would you put the chess program?
  • The biggest threat from computers comes not from intelligence but from ignorance, i.e., from computer programs that are not as smart as they need to be. I'm thinking especially of safety critical and security critical systems, such as fly-by-wire aircraft and financial trading systems. When these go wrong, aircraft crash and people are killed or the economy collapses. Worrying about intelligent machines distracts us from the real threats.
  • As far as threats go, you can't separate the machines from the intentions of their owners. Quite stupid machines entrusted to run a war with weapons of mass destruction could cause quite enough havoc without waiting for the mythical "human-level machine intelligence". It will be human owners that endow their machines with goals and aims. The less intelligent the machines the more likely this is to end in tears.
  • Given the indeterminacy of their owner's intentions, it's quite impossible to put probabilities on the questions you ask. Even if we could precisely predict the progress of the technology, which we can't, the intentions of the owners would defeat our estimates.

I'm familiar with what you call the 'standard worry'. I've frequently been recruited to public debate with Kevin Warwick, who has popularised this 'worry'. I'd be happy for you to publish my answers. I'd add one more point, which I forgot to include yesterday.

  • Consider the analogy with 'bird level flight'. Long before human flight, people aspired to fly like birds. The reality of human flight turned out to be completely different. In some respects, 'artificial' flying machines are superior to birds, e.g., they are faster. In some respects they are inferior, e.g., you have to be at the airport hours before take-off and book well in advance. The flight itself is very different, e.g., aircraft don't flap their wings. There is not much research now on flying like birds. If we really wanted to do it, we could no doubt come close, e.g., with small model birds with flapping wings, but small differences would remain and a law of diminishing returns would set in if we wanted to get closer. I think the aspiration for 'human level machine intelligence' will follow a similar trajectory --- indeed, it already has.

[Template] Questions regarding possible risks from artificial intelligence

7 XiXiDu 10 January 2012 11:59AM

I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI. Below are some questions I am going to ask. Please help to refine the questions or suggest new and better questions.

(Thanks goes to paulfchristiano, Steve Rayhawk and Mafred.)

Q1: Assuming beneficially political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans at science, mathematics, engineering and programming?

Q2: Once we build AI that is roughly as good as humans at science, mathematics, engineering and programming, how much more difficult will it be for humans and/or AIs to build an AI which is substantially better at those activities than humans?

Q3: Do you ever expect artificial intelligence to overwhelmingly outperform humans at typical academic research, in the way that they may soon overwhelmingly outperform humans at trivia contests, or do you expect that humans will always play an important role in scientific progress?

Q4: What probability do you assign to the possibility of an AI with initially (professional) human-level competence at general reasoning (including science, mathematics, engineering and programming) to self-modify its way up to vastly superhuman capabilities within a matter of hours/days/< 5 years?

Q5: How important is it to figure out how to make superhuman AI provably friendly to us and our values (non-dangerous), before attempting to build AI that is good enough at general reasoning (including science, mathematics, engineering and programming) to undergo radical self-modification?

Q6: What probability do you assign to the possibility of human extinction as a result of AI capable of self-modification (that is not provably non-dangerous, if that is even possible)?

Q&A with experts on risks from AI #2

15 XiXiDu 09 January 2012 07:40PM

[Click here to see a list of all interviews]

I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI.

(Note: I am also asking Reinforcement Learning / Universal AI researchers, teams like the one that build IBM Watson, organisations like DARPA and various companies. Some haven't replied yet while I still have to write others.)

Nils John Nilsson is one of the founding researchers in the discipline of Artificial intelligence. He is the Kumagai Professor of Engineering, Emeritus in Computer Science at Stanford University. He is particularly famous for his contributions to search, planning, knowledge representation, and robotics. [Wikipedia] [Homepage] [Google Scholar]

Peter J. Bentley, a British author and computer scientist based at University College London. [Wikipedia] [Homepage]

David Alan Plaisted is a computer science professor at the University of North Carolina at Chapel Hill. [Wikipedia] [Homepage]

Hector Levesque is a Canadian academic and researcher in artificial intelligence. He does research in the area of knowledge representation and reasoning in artificial intelligence. [Wikipedia] [Homepage]

The Interview:

Nils Nilsson: I did look at the lesswrong.com Web page.  Its goals are extremely important!  One problem about blogs like these is that I think they are read mainly by those who agree with what they say -- the already converted.  We need to find ways to get the Fox News viewers to understand the importance of what these blogs are saying.  How do we get to them?

Before proceeding to deal with your questions, you might be interested in the following:

1. An article called "Rationally-Shaped Artificial Intelligence" by Steve Omohundro about robot "morality."   It's at:

http://selfawaresystems.com/2011/10/07/rationally-shaped-artificial-intelligence/

2.  I have a draft book on "beliefs" that deals with (among other things) how to evaluate them. See:

http://ai.stanford.edu/~nilsson/Beliefs.html

(Comments welcome!)

3. Of course, you must know already about David Kahneman's excellent book, "Thinking, Fast and Slow."

Here are some (maybe hasty!) answers/responses to your questions.

David Plaisted: Your questions are very interesting and I've had such questions for a long time, actually.  I've been surprised that people are not more concerned about such things.

However, right now the problems with AI seem so difficult that I'm not worried about these issues.

Peter J. Bentley: I think intelligence is extremely hard to define, even harder to measure, and human-level intelligence is a largely meaningless phrase. All life on Earth has human-level intelligence in one sense for we have all evolved for the same amount of time and we are equally able to survive in our appropriate niches and solve problems relevant to us in highly effective ways.

There is no danger from clever AI - only from stupid AI that is so bad that it kills us by accident. I wish we were on track to create something as clever as a frog. We have a long, long way to go. I agree with Pat Hayes on this subject.

I actually find the discussion a little silly. We are *much* more likely to all become a bunch of cyborgs completely reliant on integrated tech (including some clever computers) in the next 100 years. Computers won't be external entities, they will be a part of us. Many of us already can't live our modern lives without being plugged into their gadgets for most of their waking lives. Worry about that instead :)

I think your questions are very hard to answer in any rigorous sense, for they involve prediction of future events so far ahead that anything I say is likely to be quite inaccurate. I will try to answer some below, but these are really just my educated guesses.

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

Nils Nilsson: Because human intelligence is so multi-faceted, your question really should be divided into each of the many components of intelligence. For example, on language translation, AI probably already exceeds the performance of many translators.  On integrating symbolic expressions in calculus, AI (or computer science generally) is already much better than humans.  AI does better on many planning and scheduling tasks.  On chess, same!  On the Jeopardy! quiz show, same!

A while back I wrote an essay about a replacement for the Turing test. It was called the "Employment Test."  (See:  http://ai.stanford.edu/~nilsson/OnlinePubs-Nils/General_Essays/AIMag26-04-HLAI.pdf)  How many of the many, many jobs that humans do can be done by machines?  I'll rephrase your question to be: When will AI be able to perform around 80% of these jobs as well or better than humans perform?

10% chance:  2030
50% chance:  2050
90% chance: 2100

David Plaisted: It seems that the development of human level intelligence is always later than people think it will be.  I don't have an idea how long this might take.

Peter J. Bentley: That depends on what you mean by human level intelligence and how it is measured. Computers can already surpass us at basic arithmetic. Some machine learning methods can equal us in recognition of patterns in images. Most other forms of "AI" are tremendously bad at tasks we perform well. The human brain is the result of a several billion years of evolution at molecular scales to macro scales. Our evolutionary history spans unimaginable numbers of generations, challenges, environments, predators, etc. For an artificial brain to resemble ours, it must necessarily go through a very similar evolutionary history. Otherwise it may be a clever machine, but its intelligence will be in areas that do not necessarily resemble human intelligence.

Hector Levesque: No idea. There's a lot of factors beyond wars etc mentioned. It's tough to make these kind of predictions.

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Explanatory remark to Q2:

P(human extinction | badly done AI) = ?

(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)

Nils Nilsson: 0.01% probability during the current century.  Beyond that, who knows?

David Plaisted: I think people will be so concerned about the misuse of intelligent computers that they will take safeguards to prevent such problems.  To me it seems more likely that disaster will come on the human race from nuclear or biological weapons, or possibly some natural disaster.

Peter J. Bentley: If this were ever to happen, it is most likely to be because the AI was too stupid and we relied on it too much. It is *extremely* unlikely for any AI to become "self aware" and take over the world as they like to show in the movies. It's more likely that your pot plant will take over the world.

Hector Levesque: Low. The probability of human extinction by other means (e.g. climate problems, micro biology etc) is sufficiently higher that if we were to survive all of them, surviving the result of AI work would be comparatively easy.

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?

Nils Nilsson: I'll assume that you mean sometime during this century, and that my "employment test" is the measure of superhuman intelligence.

hours:  5%
days:    50%
<5 years:  90%

David Plaisted: This would require a lot in terms of robots being able to build hardware devices or modify their own hardware.  I suppose they could also modify their software to do this, but right now it seems like a far out possibility.

Peter J. Bentley: It won't happen. Has nothing to do with internet connections or speeds. The question is rather silly.

Hector Levesque: Good. An automated human level intelligence is achieved, it ought to be able to learn what humans know more quickly.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Explanatory remark to Q4:

How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

Nils Nilsson: Work on this problem should be ongoing, I think, with the work on AGI.  We should start, now, with "little more," and gradually expand through the "much" and "vastly" as we get closer to AGI.

David Plaisted: Yes, some kind of ethical system should be built into robots, but then one has to understand their functioning well enough to be sure that they would not get around it somehow.

Peter J. Bentley: Humans are the ultimate in killers. We have taken over the planet like a plague and wiped out a large number of existing species. "Intelligent" computers would be very very stupid if they tried to get in our way. If they have any intelligence at all, they will be very friendly. We are the dangerous ones, not them.

Hector Levesque: It's always important to watch for risks with any technology. AI technology is no different.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Explanatory remark to Q5:

What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

Nils Nilsson: I think the risk of terrorists getting nuclear weapons is a greater risk than AI will be during this century. They would certainly use them if they had them -- they would be doing the work of Allah in destroying the Great Satan.  Other than that, I think global warming and other environmental problems will have a greater negative impact than AI will have during this century.  I believe technology can save us from the risks associated with new viruses.  Bill Joy worries about nano-dust, but I don't know enough about that field to assess its possible negative impacts.  Then, of course, there's the odd meteor.  Probably technology will save us from that.

David Plaisted: There are risks with any technology, even computers as we have them now.  It depends on the form of government and the nature of those in power whether technology is used for good or evil more than on the nature of the technology itself.  Even military technology can be used for repression and persecution.  Look at some countries today that use technology to keep their people in subjection.

Peter J. Bentley: No.

Hector Levesque: See above Q2. I think AI risks are smaller than others.

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

Nils Nilsson: Not as high as it should be. Some, like Steven Omohundro, Wendell Wallach and Colin Allen ("Moral Machines: Teaching Robots Right from Wrong"), Patrick Lin ("Robot Ethics"), and Ronald Arkin ("Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture") are among those thinking and writing about these problems.  You probably know of several others.

David Plaisted: Probably not as high as it should be.

Peter J. Bentley: The whole idea is blown up out of all proportion. There is no real risk and will not be for a very long time. We are also well aware of the potential risks.

Hector Levesque: Low. Technology in the area is well behind what was predicted in the past, and so concern for risks is correspondingly low.

Q7: Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?

Nils Nilsson: Because human intelligence involves so many different abilities, I think AGI will require many different technologies with many different milestones. I don't think there is a single one.  I do think, though, that the work that Andrew Ng, Geoff Hinton, and (more popularly) Jeff Hawkins and colleagues are doing on modeling learning in the neo-cortex using deep Bayes networks is on the right track.

Thanks for giving me the opportunity to think about your questions, and I hope to stay in touch with your work!

David Plaisted: I think it depends on the interaction of many different capabilities.

Peter J. Bentley: Too many advances are needed to describe here...

Hector Levesque: Reading comprehension at the level of a 10-year old.

Q8: Are you familiar with formal concepts of optimal AI design which relate to searches over complete spaces of computable hypotheses or computational strategies, such as Solomonoff induction, Levin search, Hutter's algorithm M, AIXI, or Gödel machines?

David Plaisted: My research does not specifically relate to those kinds of questions.

Q&A with experts on risks from AI #1

29 XiXiDu 08 January 2012 11:46AM

[Click here to see a list of all interviews]

Brandon Rohrer

Sandia National Laboratories
Cited by 536

Education

PhD, Mechanical Engineering, Massachusetts Institute of Technology, 2002.
Neville Hogan, Advisor and Thesis Committee Chair.

MS, Mechanical Engineering, Massachusetts Institute of Technology, 1999.
National Science Foundation Fellowship

BS cum laude, Mechanical Engineering, Brigham Young University, 1997.
Ezra Taft Benson (BYU's Presidential) Scholarship
National Merit Scholarship

Experience

Sandia National Laboratories, Albuquerque, NM.
Principal Member of the Technical Staff, 2006 - present
Senior Member of the Technical Staff, 2002 - 2006

University of New Mexico, Albuquerque, NM.
Adjunct Assistant Professor,
Department of Electrical and Computer Engineering, 2007 - present

Homepage: sandia.gov/~brrohre/

Papers: sandia.gov/rohrer/papers.html

Google Scholar: scholar.google.com/scholar?q=Brandon+Rohrer

Tim Finin

Professor of Computer Science and Electrical Engineering, University of Maryland
Cited by 20832

Tim Finin is a Professor of Computer Science and Electrical Engineering at the University of Maryland, Baltimore County (UMBC). He has over 30 years of experience in applications of Artificial Intelligence to problems in information systems and language understanding. His current research is focused on the Semantic Web, mobile computing, analyzing and extracting information from text and online social media, and on enhancing security and privacy in information systems.

Finin received an S.B. degree in Electrical Engineering from MIT and a Ph.D. degree in Computer Science from the University of Illinois at Urbana-Champaign. He has held full-time positions at UMBC, Unisys, the University of Pennsylvania, and the MIT AI Laboratory. He is the author of over 300 refereed publications and has received research grants and contracts from a variety of sources. He participated in the DARPA/NSF Knowledge Sharing Effort and helped lead the development of the KQML agent communication language and was a member of the W3C Web Ontology Working Group that standardized the OWL Semantic Web language.

Finin has chaired of the UMBC Computer Science Department, served on the board of directors of the Computing Research Association, been a AAAI councilor, and chaired several major research conferences. He is currently an editor-in-chief of the Elsevier Journal of Web Semantics.

Homepage: csee.umbc.edu/~finin/

Google Scholar: scholar.google.com/scholar?q=Tim+Finin

Pat Hayes

Pat Hayes has a BA in mathematics from Cambridge University and a PhD in Artificial Intelligence from Edinburgh. He has been a professor of computer science at the University of Essex and philosophy at the University of Illinois, and the Luce Professor of cognitive science at the University of Rochester. He has been a visiting scholar at Universite de Geneve and the Center for Advanced Study in the Behavioral Studies at Stanford, and has directed applied AI research at Xerox-PARC, SRI and Schlumberger, Inc.. At various times, Pat has been secretary of AISB, chairman and trustee of IJCAI, associate editor of Artificial Intelligence, a governor of the Cognitive Science Society and president of AAAI.

Pat's research interests include knowledge representation and automatic reasoning, especially the representation of space and time; the semantic web; ontology design; image description and the philosophical foundations of AI and computer science. During the past decade Pat has been active in the Semantic Web initiative, largely as an invited member of the W3C Working Groups responsible for the RDF, OWL and SPARQL standards. Pat is a member of the Web Science Trust and of OASIS, where he works on the development of ontology standards.

In his spare time, Pat restores antique mechanical clocks and remodels old houses. He is also a practicing artist, with works exhibited in local competitions and international collections. Pat is a charter Fellow of AAAI and of the Cognitive Science Society, and has professional competence in domestic plumbing, carpentry and electrical work.

Homepage: ihmc.us/groups/phayes/

Selected research: ihmc.us/groups/phayes/wiki/a3817/Pat_Hayes_Selected_Research.html

The Interview:

Brandon Rohrer: This is an entertaining survey. I appreciate the specificity with which you've worded some of the questions. I don't have a defensible or scientific answer to any of the questions, but I've included some answers below that are wild-ass guesses. You got some good and thoughtful responses. I've been enjoying reading them. Thanks for compiling them.

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of roughly human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

Brandon Rohrer: 2032/2052/2072

Tim Finin: 20/100/200 years

Pat Hayes: I do not consider this question to be answerable, as I do not accpet this (common) notion of "human-level intelligence" as meaningful. Artificially intelligent artifacts are in some ways superhuman, and have been for many years now; but in other ways, they are sub-human, or perhaps it would be better to say, non-human. They simply differ from human intelligences, and it is inappropriate to speak of "levels" of intelligence in this way. Intelligence is too complex and multifacetted a topic to be spoken of as though it were something like sea level that can be calibrated on a simple linear scale.

If by 'human-level' you mean, the AI will be an accurate simalcrum of a human being, or perhaps a human personality (as is often envisioned in science fiction, eg HAL from "2001") my answer would be, never. We will never create such a machine intelligence, because it is probably technically close to impossible, and not technically useful (note that HAL failed in its mission through being TOO "human": it had a nervous breakdown. Bad engineering.) But mostly because we have absolutely no need to do so. Human beings are not in such short supply at resent that it makes sense to try to make artificial ones at great cost. And actual AI work, as opposed to the fantasies often woven around it by journalists and futurists, is not aiming to create such things. A self-driving car is not an artificial human, but it is likely to be a far better driver than any human, because it will not be limited by human-level attention spans and human-level response times. It will be, in these areas, super-human, just as present computers are superhuman at calculation and keeping track of large numbers of complex patterns, etc.. .

Q2: What probability do you assign to the possibility of human extinction as a result of badly done AI?

Explanatory remark to Q2:

P(human extinction | badly done AI) = ?

(Where 'badly done' = AGI capable of self-modification that is not provably non-dangerous.)

Brandon Rohrer: < 1%

Tim Finin: 0.001

Pat Hayes: Zero. The whole idea is ludicrous.

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 GB Internet connection) = ?

Brandon Rohrer: < 1%

Tim Finin: 0.0001/0.0001/0.01

Pat Hayes: Again, zero. Self-modification in any useful sense has never been technically demonstrated. Machine learning is possible and indeed is a widely used technique (no longer only in AI) but a learning engine is the same thing after it has learnt something as it was before., just as biological learners are. When we learn, we get more informed, but not more intelligent: similarly with machines.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Explanatory remark to Q4:

How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

Brandon Rohrer: No more.

Tim Finin: No.

Pat Hayes: No. There is no reason to suppose that any manufactured system will have any emotional stance towards us of any kind, friendly or unfriendly. In fact, even if the idea of "human-level" made sense, we could have a more-than-human-level super-intelligent machine, and still have it bear no emotional stance towards other entities whatsoever. Nor need it have any lust for power or political ambitions, unless we set out to construct such a thing (which AFAIK, nobody is doing.) Think of an unworldly boffin who just wants to be left alone to think, and does not care a whit for changing the world for better or for worse, and has no intentions or desires, but simply answers questions that are put to it and thinks about htings that it is asked to think about. It has no ambition and in any case no means to achieve any far-reaching changes even if it "wanted" to do so. It seems to me that this is what a super-intelligent question-answering system would be like. I see no inherent, even slight, danger arising from the presence of such a device.

Q5: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Explanatory remark to Q5:

What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

Brandon Rohrer: Evolved variants of currently existing biological viruses and bacteria.

Tim Finin: No.

Pat Hayes: No. Nanotechnology has the potential to make far-reaching changes to the actual physical environment. AI poses no such threat. Indeed, I do not see that AI itself (that is, actual AI work being done, rather than the somewhat uninformed fantasies that some authors, such as Ray Kurtzwiel, have invented) poses any serious threat to anyone.

I would say that any human-extinction type event is likely to make a serious dent in my personal goals. (But of course I am being sarcastic, as the question as posed seems to me to be ridiculous.)

When I think of the next century, say, the risk I amost concerned about is global warming and the resulting disruption to the biosphere and human society. I do not think that humans will become extinct, but I think that our current global civilization might not survive.

Q6: What is the current level of awareness of possible risks from AI, relative to the ideal level?

Brandon Rohrer: High.

Tim Finin: About right.

Pat Hayes: The actual risks are negligible: the perceived risks (thanks to the popularization of such nonsensical ideas as the "singularity") are much greater.

Q7: Can you think of any milestone such that if it were ever reached you would expect human‐level machine intelligence to be developed within five years thereafter?

Brandon Rohrer: No, but the demonstrated ability of a robot to learn from its experience in a complex and unstructured environment is likely to be a milestone on that path, perhaps signalling HLI is 20 years away.

Tim Finin: Passing a well constructed, open ended turing test.

Pat Hayes: No. There are no 'milestones' in AI. Progress is slow but steady, and there are no magic bullets.

Anonymous

The following are replies from experts who either did not answer the questions for various reasons or didn't want them to be published.

Expert 1: Sorry, I don't want to do an email interview - it is too hard to qualify comments.

Expert 2: Thanks for your inquiry - but as you note I am a roboticist and not a futurist, so I generally try to avoid speculation.

Expert 3: my firmest belief about the timeline for human-level AI is that we can't estimate it usefully. partly this is because i don't think "human level AI" will prove to be a single thing (or event) that we can point to and say "aha there it is!". instead i think there will be a series of human level abilities that are achieved. in fact some already have (though many more haven't).

(on the other hand, i think shooting for human-level AI is a good long term research goal. it doesn't need to be one thing in the end to be a good focus of work.)

another important catch, with respect to the "risk from human level AI" equation, is that i don't think human level AI immediately leads to super-human level AI. we have had many human-level human's working on AI for a long time, and haven't added up to even a single human. i don't think it's is necessarily (or even likely) the case that a human level AI would have much more luck at making itself smarter than we have been....

Expert 4: Thanks for this - fascinating questions, and I am a great supporter of probability elicitation, but only from people who are well-informed about the subject-matter!  And I am afraid this does not include me - I am sure I should know more about this, but I don't, and so am unwilling to express publicly any firm opinion.

Of course in private in a bar I may be more forthcoming!

Expert 5: Interesting questions, I'll enjoy seeing your published results!  Unfortunately, now that I work at ****** (through the acquisition of one of my companies, ******), there are policies in place that prohibit me from participating in this kind of exercise.

Expert 6: I don't think I can answer your questions in a meaningful way...

Expert 7: Thanks for your interest. I feel that this is not in the area of my primary expertise.  However, I'd refer you to ****** ( a colleague, and co-chair of the *******) who I think might be in a better position to give you current and informed answers.

Expert 8: Unfortunately, most of these questions do not have a simple answer, in my opinion, so I can't just say "five years" or whatever -- I would have to write a little essay in order to give an answer that reflects what I really believe.  For example, the concept of "roughly human-level intelligence" is a complicated one, and any simple answer would be misleading.  By some measures we're already there; by other measures, the goal is still far in the future.  And I think that the idea of a "provably friendly" system is just meaningless.

Anyway, good luck with your survey.  I'm sure you'll get simple answers from some people, but I suspect that you will find them confusing or confused.

Expert 9: Thank you for your email. I do not feel comfortable answering your questions for a public audience.

Expert 10: sorry no reply for such questions

Expert 11: I regard speculation about AI as a waste of time.  We are at an impasse: none of our current techniques seems likely to provide truly human-like intelligence.  I think what's needed is a conceptual breakthrough from someone comparable to Newton or Einstein. Until that happens, we're going to remain stuck, although there will be lots of useful technology coming along.  It won't be "intelligent" or "conscious" the way humans are, but it might do a really good job of guessing what movies we want to watch or what news stories interest us the most.

Given our current state of ignorance, I feel that speculating about either the timeline or the impact of AI is best left to science fiction writers.

More interviews forthcoming (hopefully). At least one person told me that the questions are extremely important and that he would work out some answers over the next few days.

View more: Prev | Next