I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

In a couple of days, we are hosting a seminar in Århus (Denmark) on AI Risk.

[Survey Taken Thread]

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.

Let's make these comments a reply to this post. That way we continue the tradition, but keep the discussion a bit cleaner.

I am literally pregnant right now and wasn't sure how to answer the ones about how many children I have or if I plan more. (I went with "one" and "uncertain" but could have justified "zero" and "yes").

I am not sure the OP had much meaning behind his "vaguely magical" expression, but given that we are discussing it anyway :-) I would probably reinterpret it in terms of Knightian uncertainty. It's not only the case that we don't know, we don't know what we don't know and how much we don't know.

We have no clear idea of how humanity will travel to the stars, but the subject is neither "vaguely magical", nor is it true that the sentence "humans will visit the stars" does not refer to anything.

We have no clear idea if or how humanity will travel to the stars. I feel that discussions of things like interstellar starship engines at the moment are "vaguely magical" since no known technology suffices and it's not a "merely engineering" problem. Do you think it's useful to work on safety of interstellar engines? They could blow up and destroy a whole potential colony...

It's "vaguely magical" in sense that there is a large gap between what we have now and (U)FAI. We have no clear idea of how that gap could be crossed, we just wave hands and say "and then magic happens and we arrive at our destination".

Could we stretch the analogy to claim 3, and call some increases in human numbers "super"?

I don't know -- it all depends on what you consider "super" :-) Populations of certain organisms oscillate with much greater magnitude than humans -- see e.g. algae blooms.

Naively, the required condition is v + dH > c, where v is the velocity of the spaceship, d is the distance from the threat and H is Hubble's constant.

However, when discussing distances on the order of billions of light years and velocities near the speed of light, the complications are many, not to mention an area of current research. For a more sophisticated treatment see user Pulsar's answer to this question ...

http://physics.stackexchange.com/questions/60519/can-space-expand-with-unlimited-speed/

... in particular the graph Pulsar made for the answer ...

http://i.stack.imgur.com/Uzjtg.png

... and/or the Davis and Lineweaver paper [PDF] referenced in the answer.

Ah, so you meant the accent in 3. to be on "reaches", not on "super"?

The analogy looks like this: 1. Humans multiply, they self-improve their numbers; 2. The reproduction is recursive -- the larger a generation is, the yet larger will the next one be. Absent constraints, the growth of a population is exponential.