Comment Permalink

Eliezer Yudkowsky14y140

I am not sure that you properly appreciate what happens to people when they get old. There was once a Marvin Minsky who helped write the first paper ever on "artificial intelligence". I look forward to meeting him after he comes out of cryonic suspension, but he isn't around to talk to right now.

[anonymous]14y00

I look forward to meeting him after he comes out of cryonic suspension

What do you mean by that?

3XiXiDu14y

I do, I apparently failed to make the correct inference from your remark about him being old. I didn't think he was that "old". Well, if he answers and I notice something "peculiar", I'll refrain from publishing his response. ETA I'll err on the side of caution about his response being not reflective of him.

See in context

27 Help: Writing Marvin Minsky

by XiXiDu

10th Jun 2011

5 min read

27

I want to raise awareness of risks from AI and the challenges to mitigate those risks by writing experts and asking them questions. The e-Mail below is a template. Please help me improve it and to devise more or better questions.

Dear Mr Minsky,

I am currently trying to learn more about risks from artificial intelligence [1]. In the course of this undertaking I plan to ask various experts and influencers about their opinion. Consequently I am curious about your opinion as a noted author and cognitive scientist in the field of artificial intelligence. But first I want to apologize if I intrude on your privacy, it is not my intention to offend you or to steal your time. If that is the case, please just ignore the rest of this e-Mail.

One of the leading textbooks in artificial intelligence, 'AI: A Modern Approach' [2], states:

Omohundro (2008) hypothesizes that even an innocuous chess program could pose a risk to society. Similarly, Marvin Minsky once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers to help achieve its goal. The moral is that even if you only want you program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards.

In this regard I would like to draw your attention to the Singularity Institute for Artificial Intelligence (SIAI) [3] and their mission to solve the problem of Friendly AI [4]. One example of the research interests of the SIAI is a reflective decision theory [5] of self-modifying decision systems. The SIAI does believe that "it is one of the many fundamental open problems required to build a recursively self-improving [6] Artificial Intelligence with a stable motivational system." [7]

With this in mind, I would like to ask you the following questions:

Do you agree that risks from artificial intelligence have to be taken very seriously?
Is it important to raise awareness of those risks within the artificial intelligence community?
Should we figure out how to make AI provably friendly (non-dangerous [9]), before attempting to solve artificial general intelligence?
How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?
What probability do you assign to the possibility of us being wiped out by badly done AI?
What probability do you assign to the possibility of an intelligence explosion [10]?
What probability do you assign to the possibility of a human, respectively sub-human, level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours or days?
...

Further I would also like to ask your permission to publish and discuss your possible answers on LessWrong.com [8], to estimate the public and academic awareness and perception of risks from AI and the effectiveness with which the risks are communicated. This is however completely optional to my curiosity and general interest in your answer. I will respect your decision under any circumstances and keep your opinion private if you wish. Likewise I would be pleased, instead of, or additionally to replying to this e-Mail, with a treatment of the above questions on your homepage, your personal blog or elsewhere. You got my permission to publish my name and this e-Mail in parts or completely.

Full disclosure:

I am not associated with the SIAI or any organisation concerned with research on artificial intelligence, nor do I maintain a formal academic relationship. Given the possible permission to publish your answers they will under no circumstances be used by me in an attempt to cast a damning light on you or your interests but will be exhibited neutrally as the personal opinion of an expert.

References:

[1] "Reducing long-term catastrophic risks from artificial intelligence" http://singinst.org/riskintro/index.html
[2] "AI: A Modern Approach", Chapter 26, section 26.3, (6) "The Success of AI might mean the end of the human race." http://aima.cs.berkeley.edu/
[3] "Singularity Institute for Artificial Intelligence" http://singinst.org/
[4] "Artificial Intelligence as a Positive and Negative Factor in Global Risk." http://yudkowsky.net/singularity/ai-risk
[5] Yudkowsky, Eliezer, "Timeless Decision Theory" http://singinst.org/upload/TDT-v01o.pdf
[6] "Recursive Self-Improvement" http://lesswrong.com/lw/we/recursive_selfimprovement/
[7] "An interview with Eliezer Yudkowsky", parts 1, 2 and 3
[8] "A community blog devoted to refining the art of human rationality." http://lesswrong.com/
[9] http://wiki.lesswrong.com/wiki/Paperclip_maximizer
[10] http://wiki.lesswrong.com/wiki/Intelligence_explosion

Yours sincerely,

NAME
ADDRESS

Revised Version

Dear Professor Minsky,

I am currently trying to learn more about risks from artificial intelligence. Consequently I am curious about your opinion as a noted author and cognitive scientist in the field of artificial intelligence.

I would like to ask you the following questions:

What probability do you assign to the possibility of us being wiped out by badly done AI?
What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?
Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?
What is the current level of awareness of possible risks from AI within the artificial intelligence community, relative to the ideal level?
How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?

Further I would also like to ask your permission to publish and discuss your possible answers, in order to estimate the academic awareness and perception of risks from AI, but would also be pleased, instead of, or additionally to replying to this email, with a treatment of the above questions on your homepage, your personal blog or elsewhere.

You got my permission to publish my name and this email in parts or completely.

References:

Reducing long-term catastrophic risks from artificial intelligence: http://singinst.org/riskintro/index.html
Artificial Intelligence as a Positive and Negative Factor in Global Risk: http://yudkowsky.net/singularity/ai-risk
A community blog devoted to refining the art of human rationality: http://lesswrong.com/

Please let me know if you are interested in more material related to my questions.

Yours sincerely,

NAME
ADDRESS

Second Revision

Dear Professor Minsky,

I am currently trying to learn more about possible risks from artificial intelligence. Consequently I am curious about your opinion as a noted author and cognitive scientist in the field of artificial intelligence.

I would like to ask you the following questions:

What probability do you assign to the possibility of us being wiped out by badly done AI?
What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?
Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?
What is the current level of awareness of possible risks from AI within the artificial intelligence community, relative to the ideal level?
How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?

Furthermore I would also like to ask your permission to publish and discuss your possible answers, in order to estimate the academic awareness and perception of risks from AI.

Please let me know if you are interested in third-party material that does expand on various aspects of my questions.

Yours sincerely,

NAME
ADDRESS

Personal Blog

27

New Comment

32 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:13 PM

[-]Eliezer Yudkowsky14y150

I've met Minsky. He's, well, old. I tried asking him what he thought of Bayesianism. He said he regarded it as a failed approach. I didn't try pushing any further.

[-]jmmcd14y70

Yes -- I've seen him talk a couple of times, and everyone still loves to hear him, but he's not now influential.

I also recently saw Rodney Brooks giving the standard "rapture of the nerds" answer to a singularity question. Brooks is influential, I think, so maybe a good target.

To help XiXiDu's task, we should put together a list of useful targets.

[-]XiXiDu14y60

To help XiXiDu's task, we should put together a list of useful targets.

That would be great. I don't know of many AGI researchers. I am not going to ask Hugo De Garis, we know what Ben Goertzel thinks, and there already is an interview with Peter Voss that I will have to watch first.

More on Ben Goertzel:

He recently wrote 'Why an Intelligence Explosion is Probable', but with the caveat (see the comments):

Look -- what will prevent the first human-level AGIs from self-modifying in a way that will massively increase their intelligence is a very simple thing: they won't be smart enough to do that!

Every actual AGI researcher I know can see that. The only people I know who think that an early-stage, toddler-level AGI has a meaningful chance of somehow self-modifying its way up to massive superhuman intelligence -- are people associated with SIAI.

But I have never heard any remotely convincing arguments in favor of this odd, outlier view of the easiness of hard takeoff!!!

[-]jmmcd14y10

Jurgen Schmidhuber is one possibility.

[-]XiXiDu14y10

Jurgen Schmidhuber is one possibility.

Thanks, emailed him.

[-]XiXiDu14y10

...there already is an interview with Peter Voss that I will have to watch first.

I watched it, check 9:00 (first video) for the answer on friendly AI, he seems to agree with Ben Goertzel?

ETA

More here.

[-]timtyler14y00

We have Brooks answer to many of these questions here - at 17:20.

Essentially, I think Brooks is wrong, robots are highly likely to take over. He only addresses the "standard scenario" of a Hollywood-style hostile robot takeover.

One big possibility he fails to address is a cooperative machine takeover, with the humans and the machines on the same side.

I agree with Brooks that consumer pressure will mostly create "good" robots in the short term. Consumer-related forces will drive the extraction of human preferences into machine-readable formats, much as we are seeing privacy-related preferences being addressed by companies today. Brooks doesn't really look into later future scenarios where forces applied by human consumers are relatively puny, though. There's eventually going to be a bit of a difference between a good company, and a company that is pretending to be good for PR reasons.

I agree with Brooks that a major accident is relatively unlikely. Brooks gives a feeble reason for thinking that, though - comparing an accident with a "lone guy" building a 747. That is indeed unlikely - but surely is only one of the possible accident scenarios.

Brooks is a robot guy. Those folk are not going to build intelligent machines first. They are typically too wedded to systems with slow build-test cycles. So Brooks may be a muddle about all this, but that doesn't seem too important: it isn't really his area.

[-]XiXiDu14y50

I tried asking him what he thought of Bayesianism.

Maybe this is a stupid question, but what was the context of your question and his answer? Maybe he meant that Bayesianism isn't the answer to the problem of artificial general intelligence. I honestly can't tell if that is stupid, but does it necessarily mean that he thought Bayesianism to be a failed approach in any other context, or that it isn't part of a much larger solution?

Anyway, he is just one person. I would only have to change the template slightly to use it to email others as well.

If I get a response, I might send him another email asking about Bayesianism.

[-]Eliezer Yudkowsky14y140

[-]XiXiDu14y30

I am not sure that you properly appreciate what happens to people when they get old.

I do, I apparently failed to make the correct inference from your remark about him being old. I didn't think he was that "old". Well, if he answers and I notice something "peculiar", I'll refrain from publishing his response.

ETA

I'll err on the side of caution about his response being not reflective of him.

[-][anonymous]14y00

I look forward to meeting him after he comes out of cryonic suspension

What do you mean by that?

[-]Benquo14y130

Also, I think the last sentence is a case of protest too much:

Given the possible permission to publish your answers they will under no circumstances be used by me in an attempt to cast a damning light on you or your interests but will be exhibited neutrally as the personal opinion of an expert.

I would suggest either omitting this one entirely, or rewriting it to focus on the nice thing you will do (exhibit his response neutrally as the personal opinion of an expert) rather than the mean thing you won't do (twist his words to attack him).

[-]Manfred14y120

I'm glad someone is doing this. Upvoted.

[big quote, paragraph about SIAI] With this in mind [...]

Why? Why tell him all that, and then ask him to keep it in mind? For priming purposes? I'd prefer not to get primed estimates. For informative purposes? I don't see what's there that he wouldn't already know and that would affect his probability estimates independently of priming. The big quote and following paragraph could be condensed into a sentence or two without losing much, and gaining clarity, particularly clarity of purpose.

I agree that you should drop question 1. It's vague because of undefined values of "risks" and "seriously," and leading because people don't want a stranger to think they're "not taking risks seriously."

You should also expand question 2. What is the current level of awareness relative to the ideal level? What sort of exposure to the idea of risks do people get? Basically, I think there are similar questions with more interesting answers.

Questions 4, 5, 6 and 7 feel out of causal order. Maybe ask some questions like 6 or 7 before 4 and 5 e.g. "How likely is it that a self-modifying AGI will, sometime in the future, increase its intelligence from sub-human to superhuman?" "What would you estimate the timescale to be?"

Kudos for the LessWrong reference :P

[-]Morendil14y80

Comes across as somewhat excessively groveling; I'd suggest that the appropriate amount of groveling is to address him as "Dr Marvin Minsky" and cut the apology.

You might want to state who the other "experts and influencers" are that you plan to interview: if anything is likely to cause offense, that's what I'd bet on, so it's best got out of the way soonest.

[-]lukeprog14y70

Gert-Jan Lokhorst
Ronald Arkin
Cassio Penachin
Hugo de Garis
Lukasz Kaiser
Vladimir Redko
Keith Hoyes
Neil Sharkey
P.W. Singer (Wired for War)
Christian Lebiere
David Reitter
Thomas Metzinger
Daniel Dennett
Douglas Lenat
Ross D. King
Ali Reza Honarvar
James Moor
Michael Nielsen (quantum computation)

[-]XiXiDu14y20

What do you suggest I do with a response that doesn't state if I am allowed to publish their opinion one way or the other, shall I write another email asking explicitly about it? It feels a bit awkward bothering people like Daniel Dennett that much.

[-]Benquo14y60

I'd suggest trimming or eliminating the citation-style links (they seem a bit overwrought for a personal letter, or really anything aside from a full-blown article, maybe you can just provide 2 or 3 links and an offer of more if he's interested). Since this is a Hansonian problem of getting noticed, you might want to preferentially retain the links that connote higher status.

You also probably want to say "You have my permission" rather than "You got my permission".

And I think "e-Mail" is now deprecated in favor of "email" (though I think "e-mail" is still acceptable).

[-]Perplexed14y50

Good project. Upvoted.

[Question from list] What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?

I think you are letting your own skepticism leak into your research here. The usual FOOM scenario starts from an AI that is already superhuman (say 2x or 10x human IQ) and allows weeks or months before it reaches the irreversible stage. In the Yudkowsky/Hanson debate, the figure of two years was most frequently used as the demarcation number distinguishing FOOM from something less dramatic.

[-]XiXiDu14y00

I think you are letting your own skepticism leak into your research here. The usual FOOM scenario starts from an AI that is already superhuman (say 2x or 10x human IQ)...

No, I simply haven't been able to read the AI FOOM debate, or any of the documents that talk about FOOM, so far. I simply made that inference from my perception of people's opinion about Ben Goertzel's approach of building a toddler AGI to learn more about the nature of intelligence by gathering empirical evidence. As he writes himself:

Look -- what will prevent the first human-level AGIs from self-modifying in a way that will massively increase their intelligence is a very simple thing: they won't be smart enough to do that!

Every actual AGI researcher I know can see that. The only people I know who think that an early-stage, toddler-level AGI has a meaningful chance of somehow self-modifying its way up to massive superhuman intelligence -- are people associated with SIAI.

But I have never heard any remotely convincing arguments in favor of this odd, outlier view of the easiness of hard takeoff!!!

Additionally I posted the questions here, asking for possible corrections and improvements. I do not deliberately sneak in my personal ignorance or interests when I argue or write in the name of third-party beliefs. Just see my post 'References & Resources for LessWrong', I would never attempt to use such a project to propagate my personal opinion.

[-]jsteinhardt14y50

I think the references section at the end should be cut out; citing material that is both conceptually basic and not peer-reviewed is fairly patronizing given Minsky's status in the field.

Additionally, the questions by the end become fairly open-ended. Most professors are inundated with e-mails, you will maximize your chance of a response if you make your e-mail easy to reply to (though if you would be unsatisfied with less information than you currently ask for, I guess you can hope to get lucky).

[-]XiXiDu14y10

I added another revision to the bottom of the post and removed the references as suggested, but kept the questions.

[-]timtyler14y30

Recentish Minsky interview, with 150 questions!

[-]XiXiDu14y30

I added a revision to the post.

[-]Morendil14y50

Much better! One remaining nitpick - "You got my permission" is this time somewhat too informal; "you have my express permission" would be better.

[-]Benquo14y30

In your list of questions, maybe add one between 3 and 4 asking for alternate ways (that is, aside from provable friendliness) to decrease AI risk.

[-]Thomas14y20

I suggest you to also ask these questions everybody reading this site. In a separate thread, of course.

[-]falenas10814y10

Having both questions 1 and 2 seems redundant; I would suggest eliminating the first one. I would also be hesitant to include more questions, as with more questions he will be less inclined to answer, and any answer he does give will take longer to write.

[-][anonymous]14y10

This sentence seems like a case of protest too much:

I would suggest rewriting to make it about what you will do (exhibit the response in whole as the personal opinion of an expert) rather than what you won't do (twist his words to attack him).

[-]timtyler14y00

I'll have a bash at these questions - for reference purposes. Others may want to too.

1) What probability do you assign to the possibility of us being wiped out by badly done AI?

All humans? Less than 1%. Some due to faith in engineers. Some due to thinking that preserving some humans has substantial Universal Instrumental Value.

2) What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?

Less than 1%.

3) Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

We should put some energy into this area - though the world won't end if we don't. Machine intelligence is an enormous and important task, so the more foresight the better. I don't like this question much - the bit about being "provably friendly" frames the actually issues in this area pretty poorly.

4) What is the current level of awareness of possible risks from AI within the artificial intelligence community, relative to the ideal level?

That's mostly public information. Opinions range from blazee lack of concern through indifference (usually due to it being too far off), to powerful paranoia (from the END OF THE WORLD merchants). I'm not sure there is such a thing as an ideal level of paranoia - a spread probably provides some healthy diversity. Plus optimal paranoia levels are value-dependent.

5) How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?

Machine intelligence and nanotechnology will probably spiral together - due to G-N-R "convergence". However machine intelligence will probably lead to nanotechnology - more than the other way around. So, these risks are pretty linked together. However: machine intelligence is generally the biggest issue we face - what should get the most attention, and what could potentially cause the biggest problems if it does not go well.

[-]XiXiDu14y00

What would be an appropriate subject line?

[-]jmmcd14y00

Do you agree that risks from artificial intelligence have to be taken very seriously?

How do risks from AI compare to other existential risks, e.g. advanced nanotechnology?

The second of these implies that you're interested in existential risks, but the first should state that explicitly. Otherwise most people will interpret "risks from artificial intelligence" to include things like people losing their jobs.

Previous posts worth mentioning:

http://lesswrong.com/r/discussion/lw/4rx/singularity_and_friendly_ai_in_the_dominant_ai/

http://lesswrong.com/lw/2zv/nils_nilssons_ai_history_the_quest_for_artificial/2wc8

[-][anonymous]14y00

I'd also suggest trimming or eliminating the citation-style links (they seem a bit overwrought for a personal letter, or really anything aside from a full-blown article, maybe you can just provide 2 or 3 links and an offer of more if he's interested). Since this is a Hansonian problem of getting noticed, you might want to preferentially retain the links that connote higher status.

You also probably want to say "You have my permission" rather than "You got my permission".

And I think "e-Mail" is now deprecated in favor of "email" (though I think e-mail is still acceptable).

Moderation Log