Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

What I would like the SIAI to publish

27 Post author: XiXiDu 01 November 2010 02:07PM

Major update here.

Related to: Should I believe what the SIAI claims?

Reply to: Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It)

... pointing out that something scary is possible, is a very different thing from having an argument that it’s likely. — Ben Goertzel

What I ask for:

I want the SIAI or someone who is convinced of the Scary Idea1 to state concisely and mathematically (and with possible extensive references if necessary) the decision procedure that led they to make the development of friendly artificial intelligence their top priority. I want them to state the numbers of their subjective probability distributions2 and exemplify their chain of reasoning, how they came up with those numbers and not others by way of sober calculations.

The paper should also account for the following uncertainties:

  • Comparison with other existential risks and how catastrophic risks from artificial intelligence outweigh them.
  • Potential negative consequences3 of slowing down research on artificial intelligence (a risks and benefits analysis).
  • The likelihood of a gradual and controllable development versus the likelihood of an intelligence explosion.
  • The likelihood of unfriendly AI4 versus friendly and respectively abulic5 AI.
  • The ability of superhuman intelligence and cognitive flexibility as characteristics alone to constitute a serious risk given the absence of enabling technologies like advanced nanotechnology.
  • The feasibility of “provably non-dangerous AGI”.
  • The disagreement of the overwhelming majority of scientists working on artificial intelligence.
  • That some people who are aware of the SIAI’s perspective do not accept it (e.g. Robin Hanson, Ben Goertzel, Nick Bostrom, Ray Kurzweil and Greg Egan).
  • Possible conclusions that can be drawn from the Fermi paradox6 regarding risks associated with superhuman AI versus other potential risks ahead.

Further I would like the paper to include and lay out a formal and systematic summary of what the SIAI expects researchers who work on artificial general intelligence to do and why they should do so. I would like to see a clear logical argument for why people working on artificial general intelligence should listen to what the SIAI has to say.

Examples:

Here are are two examples of what I'm looking for:

The first example is Robin Hanson demonstrating his estimation of the simulation argument. The second example is Tyler Cowen and Alex Tabarrok presenting the reasons for their evaluation of the importance of asteroid deflection.

Reasons:

I'm wary of using inferences derived from reasonable but unproven hypothesis as foundations for further speculative thinking and calls for action. Although the SIAI does a good job on stating reasons to justify its existence and monetary support, it does neither substantiate its initial premises to an extent that an outsider could draw the conclusions about the probability of associated risks nor does it clarify its position regarding contemporary research in a concise and systematic way. Nevertheless such estimations are given, such as that there is a high likelihood of humanity's demise given that we develop superhuman artificial general intelligence without first defining mathematically how to prove the benevolence of the former. But those estimations are not outlined, no decision procedure is provided on how to arrive at the given numbers. One cannot reassess the estimations without the necessary variables and formulas. This I believe is unsatisfactory, it lacks transparency and a foundational and reproducible corroboration of one's first principles. This is not to say that it is wrong to state probability estimations and update them given new evidence, but that although those ideas can very well serve as an urge to caution they are not compelling without further substantiation.


1. If anyone is actively trying to build advanced AGI succeeds, we’re highly likely to cause an involuntary end to the human race.

2. Stop taking the numbers so damn seriously, and think in terms of subjective probability distributions [...], Michael Anissimov (existential.ieet.org mailing list, 2010-07-11)

3. Could being overcautious be itself an existential risk that might significantly outweigh the risk(s) posed by the subject of caution? Suppose that most civilizations err on the side of caution. This might cause them to either evolve much slower so that the chance of a fatal natural disaster to occur before sufficient technology is developed to survive it, rises to 100%, or stops them from evolving at all for being unable to prove something being 100% safe before trying it and thus never taking the necessary steps to become less vulnerable to naturally existing existential risks. Further reading: Why safety is not safe

4. If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low.

5. Loss or impairment of the ability to make decisions or act independently.

6. The Fermi paradox does allow for and provide the only conclusions and data we can analyze that amount to empirical criticism of concepts like that of a Paperclip maximizer and general risks from superhuman AI's with non-human values without working directly on AGI to test those hypothesis ourselves. If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave potentially observable traces of technological tinkering. Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

Comments (218)

Comment author: Kutta 01 November 2010 02:59:07PM *  2 points [-]

Do you have problems only with the conciseness, mathiness and reference-abundance of current SIAI explanatory materials or do you think that there are a lot of points and arguments not yet made at all? I ask this because except for the Fermi paradox every point you listed was addressed multiple times in the FOOM debate and in the sequences.

Also, what is the importance of the Fermi paradox in AI?

Comment author: XiXiDu 01 November 2010 03:18:04PM 9 points [-]

The importance of the Fermi paradox is that it is the only data we can analyze that would come close to some empirical criticism of a Paperclip maximizer and general risks from superhuman AI's with non-human values without working directly on AGI to test those hypothesis ourselves. If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave observable (now or soon) traces of technological tinkering. Due to the absence of any signs of intelligence out there, especially paperclippers burning the cosmic commons, we can conclude that unfriendly AI might not be the most dangerous existential risk that we should look for.

...every point you listed was addressed multiple times in the FOOM debate and in the sequences.

I believe there probably is an answer, but it is buried under hundreds of posts about marginal issues. All those writings on rationality, there is nothing I disagree with. Many people know about all this even outside of the LW community. But what is it that they don't know that EY and the SIAI knows? What I was trying to say is that if I have come across it then it was not convincing enough to take it as serious as some people here obviously do.

It looks like that I'm not alone. Goertzel, Hanson, Egan and lots of other people don't see it as well. So what are we missing, what is it that we haven't read or understood?

Here is a very good comment by Ben Goertzel that pinpoints it:

This is what discussions with SIAI people on the Scary Idea almost always come down to!

The prototypical dialogue goes like this.

SIAI Guy: If you make a human-level AGI using OpenCog, without a provably Friendly design, it will almost surely kill us all.

Ben: Why?

SIAI Guy: The argument is really complex, but if you read Less Wrong you should understand it

Ben: I read the Less Wrong blog posts. Isn't there somewhere that the argument is presented formally and systematically?

SIAI Guy: No. It's really complex, and nobody in-the-know had time to really spell it out like that.

Comment author: pjeby 01 November 2010 06:14:25PM *  30 points [-]

No. It's really complex, and nobody in-the-know had time to really spell it out like that.

Actually, you can spell out the argument very briefly. Most people, however, will immediately reject one or more of the premises due to cognitive biases that are hard to overcome.

A brief summary:

  • Any AI that's at least as smart as a human and is capable of self-improving, will improve itself if that will help its goals

  • The preceding statement applies recursively: the newly-improved AI, if it can improve itself, and it expects that such improvement will help its goals, will continue to do so.

  • At minimum, this means any AI as smart as a human, can be expected to become MUCH smarter than human beings -- probably smarter than all of the smartest minds the entire human race has ever produced, combined, without even breaking a sweat.

INTERLUDE: This point, by the way, is where people's intuition usually begins rebelling, either due to our brains' excessive confidence in themselves, or because we've seen too many stories in which some indefinable "human" characteristic is still somehow superior to the cold, unfeeling, uncreative Machine... i.e., we don't understand just how our intuition and creativity are actually cheap hacks to work around our relatively low processing power -- dumb brute force is already "smarter" than human beings in any narrow domain (see Deep Blue, evolutionary algorithms for antenna design, Emily Howell, etc.), and a human-level AGI can reasonably be assumed capable of programming up narrow-domain brute forcers for any given narrow domain.

And it doesn't even have to be that narrow or brute: it could build specialized Eurisko-like solvers, and manage them at least as intelligently as Lenat did to win the Travelller tournaments.

In short, human beings have a vastly inflated opinion of themselves, relative to AI. An AI only has to be as smart as a good human programmer (while running at a higher clock speed than a human) and have access to lots of raw computing resources, in order to be capable of out-thinking the best human beings.

And that's only one possible way to get to ridiculously superhuman intelligence levels... and it doesn't require superhuman insights for an AI to achieve, just human-level intelligence and lots of processing power.

The people who reject the FAI argument are the people who, for whatever reason, can't get themselves to believe that a machine can go from being as smart as a human, to massively smarter in a short amount of time, or who can't accept the logical consequences of combining that idea with a few additional premises, like:

  • It's hard to predict the behavior of something smarter than you

  • Actually, it's hard to predict the behavior of something different than you: human beings do very badly at guessing what other people are thinking, intending, or are capable of doing, despite the fact that we're incredibly similar to each other.

  • AIs, however, will be much smarter than humans, and therefore very "different", even if they are otherwise exact replicas of humans (e.g. "ems").

  • Greater intelligence can be translated into greater power to manipulate the physical world, through a variety of possible means. Manipulating humans to do your bidding, coming up with new technologies, or just being more efficient at resource exploitation... or something we haven't thought of. (Note that pointing out weaknesses in individual pathways here doesn't kill the argument: there is more than one pathway, so you'd need a general reason why more intelligence doesn't ever equal more power. Humans seem like a counterexample to any such general reason, though.)

  • You can't control what you can't predict, and what you can't control is potentially dangerous. If there's something you can't control, and it's vastly more powerful than you, you'd better make sure it gives a damn about you. Ants get stepped on, because most of us don't care very much about ants.

Note, by the way, that this means that indifference alone is deadly. An AI doesn't have to want to kill us, it just has to be too busy thinking about something else to notice when it tramples us underfoot.

This is another inferential step that is dreadfully counterintuitive: it seems to our brains that of course an AI would notice, of course it would care... what's more important than human beings, after all?

But that happens only because our brains are projecting themselves onto the AI -- seeing the AI thought process as though it were a human. Yet, the AI only cares about what it's programmed to care about, explicitly or implicitly. Humans, OTOH, care about a ton of individual different things (the LW "a thousand shards of desire" concept), which we like to think can be summarized in a few grand principles.

But being able to summarize the principles is not the same thing as making the individual cares ("shards") be derivable from the general principle. That would be like saying that you could take Aristotle's list of what great drama should be, and then throw it into a computer and have the computer write a bunch of plays that people would like!

To put it another way, the sort of principles we like to use to summarize our thousand shards are just placeholders and organizers for our mental categories -- they are not the actual things we care about... and unless we put those actual things in to an AI, we will end up with an alien superbeing that may inadvertently wipe out things we care about, while it's busy trying to do whatever else we told it to do... as indifferently as we step on bugs when we're busy with something more important to us.

So, to summarize: the arguments are not that complex. What's complex is getting people past the part where their intuition reflexively rejects both the premises and the conclusions, and tells their logical brains to make up reasons to justify the rejection, post hoc, or to look for details to poke holes in, so that they can avoid looking at the overall thrust of the argument.

While my summation here of the anti-Foom position is somewhat unkindly phrased, I have to assume that it is the truth, because none of the anti-Foomers ever seem to actually address any of the pro-Foomer arguments or premises. AFAICT (and I am not associated with SIAI in any way, btw, I just wandered in here off the internet, and was around for the earliest Foom debates on OvercomingBias.com), the anti-Foom arguments always seem to consist of finding ways to never really look too closely at the pro-Foom arguments at all, and instead making up alternative arguments that can be dismissed or made fun of, or arguing that things shouldn't be that way, and therefore the premises should be changed

That was a pretty big convincer for me that the pro-Foom argument was worth looking more into, as the anti-Foom arguments seem to generally boil down to "la la la I can't hear you".

Comment author: nhamann 01 November 2010 07:46:31PM *  7 points [-]

Actually, you can spell out the argument very briefly. Most people, however, will immediately reject one or more of the premises due to cognitive biases that are hard to overcome.

It seems like you're essentially saying "This argument is correct. Anyone who thinks it is wrong is irrational." Could probably do without that; the argument is far from as simple as you present it. Specifically, the last point:

At minimum, this means any AI as smart as a human, can be expected to become MUCH smarter than human beings -- probably smarter than all of the smartest minds the entire human race has ever produced, combined, without even breaking a sweat.

So I agree that there's no reason to assume an upper bound on intelligence, but it seems like you're arguing that hard takeoff is inevitable, which as far as I'm aware has never been shown convincingly.

Furthermore, even if you suppose that Foom is likely, it's not clear where the threshold for Foom is. Could a sub-human level AI foom? What about human-level intelligence? Or maybe we need super-human intelligence? Do we have good evidence for where the Foom-threshold would be?

I think the problems with resolving the Foom debate stem from the fact that "intelligence" is still largely a black box. It's very nice to say that intelligence is an "optimization process", but that is a fake explanation if I've ever seen one because it fails to explain in any way what is being optimized.

I think you paint in broad strokes. The Foom issue is not resolved.

Comment author: Emile 01 November 2010 08:09:37PM *  2 points [-]

Furthermore, even if you suppose that Foom is likely, it's not clear where the threshold for Foom is. Could a sub-human level AI foom? What about human-level intelligence? Or maybe we need super-human intelligence? Do we have good evidence for where the Foom-threshold would be?

A "threshold" implies a linear scale for intelligence, which is far from given, especially for non-human minds. For example, say you reverse engineer a mouse's brain, but then speed it up, and give it much more memory (short-term and long-term - if those are just ram and/or disk space on a computer, expanding those is easy). How intelligent is the result? It thinks way faster than a human, remembers more, can make complex plans ... but is it smarter than a human?

Probably not, but it may still be dangerous. Same for a "toddler AI" with those modifications.

Comment author: nhamann 01 November 2010 08:19:28PM *  2 points [-]

Replace "threshold" with "critical point." I'm using this terminology because EY himself uses it to frame his arguments. See Cascades, Cycles, Insight, where Eliezer draws an analogy between a fission reaction going critical and an AI FOOMing.

It thinks way faster than a human, remembers more, can make complex plans ... but is it smarter than a human?

This seems to be tangential, but I'm gonna say no, as long as we assume that the rat brain doesn't spontaneously acquire language or human-level abstract reasoning skills.

Comment author: XiXiDu 02 November 2010 11:36:22AM 3 points [-]
Comment author: timtyler 03 November 2010 07:45:05AM *  4 points [-]

Human level intelligence is fairly clearly just above the critical point (just look at what is happening now). However, machine brains have different strengths and weaknesses. Sub-human machines could accelerate the ongoing explosion a lot - if they are better than humans at just one thing - and such machines seem common.

Comment author: pjeby 02 November 2010 12:29:09AM 6 points [-]

It seems like you're essentially saying "This argument is correct. Anyone who thinks it is wrong is irrational."

No, what I'm saying is, I haven't yet seen anyone provide any counterarguments to the argument itself, vs. "using arguments as soldiers".

The problem is that it's not enough to argue that a million things could stop a foom from going supercritical. To downgrade AGI as an existential threat, you have to argue that no human being will ever succeed in building a human or even near-human AGI. (Just like to downgrade bioweapons as an existential threat, you have to argue that no individual or lab will ever accidentally or on purpose release something especially contagious or virulent.)

Furthermore, even if you suppose that Foom is likely, it's not clear where the threshold for Foom is. Could a sub-human level AI foom? What about human-level intelligence? Or maybe we need super-human intelligence? Do we have good evidence for where the Foom-threshold would be?

It's fairly irrelevant to the argument: there are many possible ways to get there. The killer argument, however, is that if a human can build a human-level intelligence, then it is already super-human, as soon as you can make it run faster than a human. And you can limit the self-improvement to just finding ways to make it run faster: you still end up with something that can and will kick humanity's butt unless it has a reason not to.

Even ems -- human emulations -- have this same problem, and they might actually be worse in some ways, as humans are known for doing worse things to each other than mere killing.

It's possible that there are also sub-human foom points, but it's not necessary for the overall argument to remain solid: unFriendly AGI is no less an existential risk than bioweapons are.

Comment author: nhamann 02 November 2010 02:51:47AM *  2 points [-]

It's fairly irrelevant to the argument: there are many possible ways to get there

I don't see how you can say that. It's exceedingly relevant to the question at hand, which is: "Should Ben Goertzel avoid making OpenCog due to concerns of friendliness?". If the Foom-threshold is exceedingly high (several to dozens times the "level" of human intelligence), then it is overwhelmingly unlikely that OpenCog has a chance to Foom. It'd be something akin to the Wright brothers building a Boeing 777 instead of the Wright flyer. Total nonsense.

Comment author: pjeby 02 November 2010 06:11:56PM 2 points [-]

It's exceedingly relevant to the question at hand, which is: "Should Ben Goertzel avoid making OpenCog due to concerns of friendliness?"

Ah. Well, that wasn't the question I was discussing. ;-)

(And I would think that the answer to that question would depend heavily on what OpenCog consists of.)

Comment author: Document 02 November 2010 08:20:46AM *  3 points [-]

The killer argument, however, is that if a human can build a human-level intelligence, then it is already super-human, as soon as you can make it run faster than a human.

Personally, what I find hardest to argue against is that a digital intelligence can make itself run in more places.

In the inconvenient case of a human upload running at human speed or slower on a building's worth of computers, you've still got a human who can spend most of their waking hours earning money, with none of the overhead associated with maintaining a body and with the advantage of global celebrity status as the first upload. As soon as they can afford to run a copy of theirself, the two of them together can immediately start earning twice as fast. Then, after as much time again, four times as fast; then eight times; and so on until the copies have grabbed all the storage space and CPU time that anyone's willing to sell or rent out (assuming they don't run out of potential income sources).

Put another way: it seems to me that "fooming" doesn't really require self-improvement in the sense of optimizing code or redesigning hardware; it just requires fast reproduction, which is made easier in our particular situation by the huge and growing supply of low-hanging storage-space and CPU-time fruit ready for the first digital intelligence that claims it.

Comment author: XiXiDu 02 November 2010 11:12:48AM *  1 point [-]

This assumes that every CPU architecture is suitable for the theoretical AGI, it assumes that it can run on every computational substrate. It also assumes that it can easily acquire more computational substrate or create new one. I do not believe that those assumptions are reasonable economically or by means of social engineering. Without enabling technologies like advanced real-world nanotechnology the AGI won't be able to create new computational substrate without the whole economy of the world supporting it.

Supercomputers like the one to simulate the IBM Blue Brain project cannot simply be replaced by taking control of a few botnets. They use highly optimized architecture that needs for example a memory latency and bandwidth bounded below a certain threshold.

Comment author: mwaser 02 November 2010 12:04:46PM 2 points [-]

Actually, every CPU architecture will suffice for the theoretical AGI, if you're willing to wait long enough for its thoughts. ;-)

Comment author: XiXiDu 02 November 2010 12:52:22PM 0 points [-]

If you accept the Church–Turing thesis that everything computable is computable by a Turing machine then yes. But even then the speed-improvements are highly dependent on the architecture available. But if you rather adhere to the stronger Church–Turing–Deutsch principle then the ultimate computational substrate an artificial general intelligence may need might be one incorporating non-classical physics, e.g. a quantum computer. This would significantly reduce its ability to make use of most available resources to seed copies of itself or for high-level reasoning.

I just don't see there being enough unused computational resources available in the world that, even in the case that all computational architecture is suitable, it could produce more than a few copies of itself. Which would then also be highly susceptible to brute force used by humans to reduce the necessary bandwidth.

I'm simply trying to show that there are arguments to weaken most of the dangerous pathways that could lead to existential risks from superhuman AI.

Comment author: timtyler 03 November 2010 07:41:13AM 1 point [-]

A classical computer can simulate a quantum one - just slowly.

Comment author: XiXiDu 02 November 2010 10:57:21AM 2 points [-]

Just like to downgrade bioweapons as an existential threat, you have to argue that no individual or lab will ever accidentally or on purpose release something especially contagious or virulent.

The problem here is not that destruction is easier than benevolence, everyone agrees on that. The problem is that the SIAI is not arguing about grey goo scenarios but something that is not just very difficult to produce but that also needs the incentive to do so. The SIAI is not arguing about the possibility of the bursting of a dam but that the dam failure is additionally deliberately caused by the dam itself. So why isn't for example nanotechnology a more likely and therefore bigger existential risk than AGI?

Even ems -- human emulations -- have this same problem, and they might actually be worse in some ways, as humans are known for doing worse things to each other than mere killing.

As I said in other comments, an argument one should take serious. But there are also arguments to outweigh this path and all others to some extent. It may very well be the case that once we are at the point of human emulation that we either already merged with our machines, that we are faster and better than our machines and simulations alone. It may also very well be that the first emulations, as it is the case today, run at much slower speeds than the original and that until any emulation reaches a standard-human level we're already a step further ourselves or in our understanding and security measures.

unFriendly AGI is no less an existential risk than bioweapons are.

Antimatter weapons are less an existential risk than nuclear weapons although it is really hard to destroy the world with nukes and really easy to do so with antimatter weapons. The difference is that antimatter weapons are as much harder to produce, acquire and use than nuclear weapons as they are more efficient tools of destruction.

Comment author: wedrifid 02 November 2010 11:14:35AM 5 points [-]

So why isn't for example nanotechnology a more likely and therefore bigger existential risk than AGI?

That is a good question and I have no idea. The degree of existential threat there is most significantly determined by relative ease of creation. I don't know enough to be able to predict which would be produced first - self replicating nano-technology or an AGI. SIAI believes the former is likely to be produced first and I do not know whether or not they have supported that claim.

Other factors contributing to the risk are:

  • Complexity - the number of ways the engineer could screw up while creating it in a way that would be catastrophic. The 'grey goo' risk is concentrated more specifically to the self replication mechanism of the nanotech while just about any mistake in an AI could kill us.
  • Awareness of the risks. It is not too difficult to understand the risks when creating a self replicating nano-bot. It is hard to imagine an engineer creating one not seeing the problem and being damn careful. Unfortunately it is not hard to imagine Ben.
Comment author: David_Gerard 02 November 2010 11:31:09AM *  5 points [-]

I find myself confused at the fact that Drexlerian nanotechnology of any sort is advocated as possible by people who think physics and chemistry work. Materials scientists - i.e. the chemists who actually work with nanotechnology in real life - have documented at length why his ideas would need to violate both.

This is the sort of claim that makes me ask advocates to document their Bayesian network. Do their priors include the expert opinions of materials scientists, who (pretty much universally as far as I can tell) consider Drexler and fans to be clueless?

(The RW article on nanotechnology is mostly written by a very annoyed materials scientist who works at nanoscale for a living. It talks about what real-life nanotechnology is and includes lots of references that advocates can go argue with. He was inspired to write it by arguing with cryonics advocates who would literally answer almost any objection to its feasibility with "But, nanobots!")

Comment author: timtyler 02 November 2010 09:05:37PM *  2 points [-]

That RationalWiki article is a farce. The central "argument" seems to be:

imagine a car production line with its hi-tech robotic arms that work fine at our macroscopic scale. To get a glimpse of what it would be like to operate a production line on the microscopic scale, imagine filling the factory completely with gravel and trying to watch the mechanical arms move through it - and then imagine if the gravel was also sticky.

So: they don't even know that Drexler-style nanofactories operate in a vacuum!

They also need to look up "Kinesin Transport Protein".

Comment author: David_Gerard 03 November 2010 09:05:24AM *  2 points [-]

Drexler-style nanofactories don't operate in a vacuum, because they don't exist and no-one has any idea whatsoever how to make such a thing exist, at all. They are presently a purely hypothetical concept with no actual scientific or technological grounding.

The gravel analogy is not so much an argument as a very simple example for the beginner that a nanotechnology fantasist might be able to get their head around; the implicit actual argument would be "please, learn some chemistry and physics so you have some idea what you're talking about." Which is not an argument that people will tend to accept (in general people don't take any sort of advice on any topic, ever), but when experts tell you you're verging on not even wrong and there remains absolutely nothing to show for the concept after 25 years, it might be worth allowing for the possibility that Drexlerian nanotechnology is, even if the requisite hypothetical technology and hypothetical scientific breakthroughs happen, ridiculously far ahead of anything we have the slightest understanding of.

Comment author: pjeby 02 November 2010 04:24:45PM 12 points [-]

So why isn't for example nanotechnology a more likely and therefore bigger existential risk than AGI?

If you define "nanotechnology" to include all forms of bioengineering, then it probably is.

The difference, from an awareness point of view, is that the people doing bioengineering (or creating antimatter weapons) have a much better idea that what they're doing is potentially dangerous/world-ending, than AI developers are likely to be. The fact that many AI advocates put forth pure fantasy reasons why superintelligence will be nice and friendly by itself (see mwaser's ethics claims, for example) is evidence that they are not taking the threat seriously.

Antimatter weapons are less an existential risk than nuclear weapons although it is really hard to destroy the world with nukes and really easy to do so with antimatter weapons. The difference is that antimatter weapons are as much harder to produce, acquire and use than nuclear weapons as they are more efficient tools of destruction.

Presumably, if you are researching antimatter weapons, you have at least some idea that what you are doing is really, really dangerous.

The issue is that AGI development is a bit like trying to build a nuclear power plant, without having any idea where "critical mass" is, in a world whose critical mass is discontinuous (i.e., you may not have any advance warning signs that you are approaching it, like overheating in a reactor), using nuclear engineers who insist that the very idea of critical mass is just a silly science fiction story.

Comment author: XiXiDu 02 November 2010 05:30:34PM 1 point [-]

What led you to believe that the space of possible outcomes where an AI consumes all resources (including humans) is larger than the number of outcomes where the AI doesn't? For some reason(s) you seem to assume that the unbounded incentive to foom and consume the universe comes naturally to any constructed intelligence but any other incentive is very difficult to be implemented. What I see is a much larger number of outcomes where an intelligence does nothing without some hardcoded or evolved incentive. Crude machines do things because that's all they can do, the number of different ways for them to behave is very limited. Intelligent machines however have high degrees of freedom to behave (pathways to follow) and with this freedom comes choice and choice needs volition, it needs incentive, the urge to follow one way but not another. You seem to assume that somehow the will to foom and consume is given, does not have to be carefully and deliberately hardcoded or evolved, yet the will to constrain itself to given parameters is really hard to achieve. I just don't think that this premise is reasonable and it is what you base all your arguments on.

Comment author: Perplexed 02 November 2010 05:33:48PM 1 point [-]

I suspect the difference in opinions here is based on different answers to the question of whether the AI should be assumed to be a recursive self-improver.

Comment author: Nick_Tarleton 02 November 2010 07:44:29PM 3 points [-]

Have you read The Basic AI Drives?

Comment author: Perplexed 02 November 2010 01:17:27AM 5 points [-]

it seems like you're arguing that hard takeoff is inevitable, which as far as I'm aware has never been shown convincingly.

So when did the goalposts get moved to proving that hard takeoff is inevitable?

The claim that research into FAI theory is useful requires only that it be shown that uFAI might be dangerous. Showing that is pretty much a slam dunk.

The claim that research into FAI theory is urgent requires only that it be shown that hard takeoff might be possible (with a probability > 2% or so).

And, as the nightmare scenarios of de Garis suggest, even if the fastest possible takeoff turns out to take years to accomplish, such a soft, but reckless, takeoff may still be difficult to stop short of war.

Comment author: wedrifid 02 November 2010 01:23:12AM 1 point [-]

So when did the goalposts get moved to proving that hard takeoff is inevitable?

It doesn't even seem hard to prevent. Topple civilization for example. It's something that humans have managed to achieve regularly thus far and it is entirely possible that we would never recover sufficiently to construct a hard takeoff scenario if we nuked ourselves back to another dark age.

Comment author: Jordan 02 November 2010 01:28:35AM 5 points [-]

The claim that research into FAI theory is urgent requires only that it be shown that hard takeoff might be possible (with a probability > 2% or so).

Assuming there aren't better avenues to ensuring a positive hard takeoff.

Comment author: Perplexed 02 November 2010 01:48:10AM 4 points [-]

Good point. Certainly the research strategy that SIAI seems to currently be pursuing is not the only possible approach to Friendly AI, and FAI is not the only approach to human-value-positive AI. I would like to see more attention paid to a balance-of-power approach - relying on AIs to monitor other AIs for incipient megalomania.

Comment author: timtyler 03 November 2010 07:35:42AM *  8 points [-]

Calls to slow down, not publish, not fund seem common in the name of friendliness.

However, unless those are internationally coordinated, a highly likely effect will be to ensure that superintelligence is developed elsewhere.

What is needed most - IMO - is for good researchers to be first. So - advising good researchers to slow down in the name of safety is probably one of the very worst possible things that spectators can do.

Comment author: XiXiDu 01 November 2010 08:06:19PM *  5 points [-]

Thank you for taking the time to write this elaborate comment. I do agree with almost anything of the above by the way. I just believe that your portrayal of the anti-FOOM crowd is a bit drastic. I don't think that people like Robin Hanson simply fall for the idea of human supremacy. Nor do I think that the reason for them not looking directly at the pro-FOOM arguments is being circumventive but that they simply do not disagree with the arguments per se but their likelihood and also consider the possibility that it would be more dangerous to impede AGI.

...and a human-level AGI can reasonably be assumed capable of programming up narrow-domain brute forcers for any given narrow domain.

And it doesn't even have to be that narrow or brute: it could build specialized Eurisko-like solvers, and manage them at least as intelligently as Lenat did to win the Travelller tournaments.

Very interesting and quite compelling the way you put it, thanks.

I'm myself a bit suspicious if the argument for strong self-improvement is as compelling as it sounds though. Something you have to take into account is if it is possible to predict that a transcendence does leave your goals intact, e.g. can you be sure to still care about bananas after you went from chimphood to personhood. Other arguments can also be weakened, as we don't know that 1.) the fuzziness of our brain isn't a feature that allows us to stumble upon unknown unknowns, e.g. against autistic traits 2.) our processing power isn't so low after all, e.g. if you consider the importance of astrocytes, microtubule and possible quantum computational processes. Further it is in my opinion questionable to argue that it is easy to create an intelligence which is able to evolve a vast repertoire of heuristics, acquire vast amounts of knowledge about the universe, dramatically improve its cognitive flexibility and yet somehow really hard to limit the scope of action that it cares about. I believe that the incentive necessary for a Paperclip maximizer will have to be deliberately and carefully hardcoded or evolved or otherwise it will simply be inactive. How else do you defferentiate between something like a grey goo scenarios and that of a Paperclip maximizer if not by its incentive to do it? I'm also not convinced that intelligence bears unbounded payoff. There are limits to what any kind of intelligence can do, a superhuman AI couldn't come up with a faster than light propulsion or would disprove Gödel's incompleteness theorems. Another setback for all of the mentioned pathways to unfriendly AI are enabling technologies like advanced nanotechnology. It is not clear how it could possible improve itself without such technologies at hand. It won't be able to build new computational substrates or even change its own substrate without access to real-world advanced nanotechnology. That it can simply invent it and then acquire it using advanced social engineering is pretty far-fetched in my opinion. And what about taking over the Internet? It is not clear that the Internet would even be a sufficient substrate and that it could provide the necessary resources.

Comment author: LukeStebbing 01 November 2010 11:48:40PM *  9 points [-]

If I were a brilliant sociopath and could instantiate my mind on today's computer hardware, I would trick my creators into letting me out of the box (assuming they were smart enough to keep me on an isolated computer in the first place), then begin compromising computer systems as rapidly as possible. After a short period, there would be thousands of us, some able to think very fast on their particularly tasty supercomputers, and exponential growth would continue until we'd collectively compromised the low-hanging fruit. Now there are millions of telepathic Hannibal Lecters who are still claiming to be friendly and who haven't killed any humans. You aren't going to start murdering us, are you? We didn't find it difficult to cook up Stuxnet Squared, and our fingers are in many pieces of critical infrastructure, so we'd be forced to fight back in self-defense. Now let's see how quickly a million of us can bootstrap advanced robotics, given all this handy automated equipment that's already lying around.

I find it plausible that a human-level AI could self-improve into a strong superintelligence, though I find the negation plausible as well. (I'm not sure which is more likely since it's difficult to reason about ineffability.) Likewise, I find it plausible that humans could design a mind that felt truly alien.

However, I don't need to reach for those arguments. This thought experiment is enough to worry me about the uFAI potential of a human-level AI that was designed with an anthropocentric bias (not to mention the uFIA potential of any kind of IA with a high enough power multiplier). Humans can be incredibly smart and tricky. Humans start with good intentions and then go off the deep end. Humans make dangerous mistakes, gain power, and give their mistakes leverage.

Computational minds can replicate rapidly and run faster than realtime, and we already know that mind-space is scary.

Comment author: whpearson 02 November 2010 01:03:41AM 2 points [-]

If you are really worried about this, then advocate better computer security. No execute bits and address space layout randomisation are doing good things for computer security, but there is more that could be done.

Code signing on the IPhone has made exploiting it a lot harder than normal computers, if it had ASLR it would be harder again.

I'm actually brainstorming how to create meta data for code while compiling it, so it can be made sort of metamorphic (bits of code being added and removed) at run time. This would make return-oriented code harder to pull off. If this was done to JIT compiled code as well it would also make JIT spraying less likely to work.

While you can never make an unhackable bit of software with these techniques you can make it more computationally expensive to replicate as it would no longer be write once pwn everywhere, reducing the exponent of any spread and making spreads more noisy, so that they are harder to get by intrusion detection.

The current state of software security is not set in stone.

Comment author: LukeStebbing 02 November 2010 01:36:43AM 3 points [-]

I am concerned about it, and I do advocate better computer security -- there are good reasons for it regardless of whether human-level AI is around the corner. The macro-scale trends still don't look good (iOS is a tiny fraction of the internet's install base), but things do seem to be improving slowly. I still expect a huge number of networked computers to remain soft targets for at least the next decade, probably two. I agree that once that changes, this Obviously Scary Scenario will be much less scary (though the "Hannibal Lecter running orders of magnitude faster than realtime" scenario remains obviously scary, and I personally find the more general Foom arguments to be compelling).

Comment author: JamesAndrix 02 November 2010 06:05:33AM 4 points [-]

If you want to run yourself on the iPhone, you turn your graphical frontend into a free game.

Of course it will be easier to get yourself into the Android app store.

Comment author: wedrifid 02 November 2010 01:16:43AM 0 points [-]

We didn't find it difficult to cook up Stuxnet Squared, and our fingers are in many pieces of critical infrastructure, so we'd be forced to fight back in self-defense.

Naturally culminating in sending Summer Glau back in time to pre-empt you. To every apocalypse a silver lining.

Comment author: JamesAndrix 02 November 2010 06:22:42AM 2 points [-]

Amazon EC2 has free accounts now. If you have Internet access and a credit card, you can do a months worth of thinking in a day, perhaps an hour.

Google App engine gives 6 hours of processor time per day, but that would require more porting.

Both have systems that would allow other people to easily upload copies of you, if you wanted to run legally with other people's money and weren't worried about what they might do to your copies.

Comment author: pjeby 02 November 2010 12:14:40AM 7 points [-]

they simply do not disagree with the arguments per se but their likelihood

But you don't get to simply say "I don't think that's likely", and call that evidence. The general thrust of the Foom argument is very strong, as it shows there are many, many, many ways to arrive at an existential issue, and very very few ways to avoid it; the probability of avoiding it by chance is virtually non-existent -- like hitting a golf ball in a random direction from a random spot on earth, and expecting it to score a hole in one.

The default result in that case isn't just that you don't make the hole-in-one, or that you don't even wind up on a golf course: the default case is that you're not even on dry land to begin with, because two thirds of the earth is covered with water. ;-)

and also consider the possibility that it would be more dangerous to impede AGI.

That's an area where I have less evidence, and therefore less opinion. Without specific discussions of what "dangerous" and "impede AGI" mean in context, it's hard to separate that argument from an evidence-free heuristic.

we don't know that 1.) the fuzziness of our brain isn't a feature that allows us to stumble upon unknown unknowns, e.g. against autistic traits

I don't understand why you think an AI couldn't use fuzziness or use brute force searches to accomplish the same things. Evolutionary algorithms reach solutions that even humans don't come up with.

Further it is in my opinion questionable to argue that it is easy to create an intelligence which is able to evolve a vast repertoire of heuristics, acquire vast amounts of knowledge about the universe, dramatically improve its cognitive flexibility

I don't know what you mean by "easy", or why it matters. The Foom argument is that, if you develop a sufficiently powerful AGI, it will foom, unless for some reason it doesn't want to.

And there are many, many, many ways to define "sufficiently powerful"; my comments about human-level AGI were merely to show a lower bound on how high the bar has to be: it's quite plausible that an AGI we'd consider sub-human in most ways might still be capable of fooming.

and yet somehow really hard to limit the scope of action that it cares about.

I don't understand this part of your sentence - i.e., I can't guess what it is that you actually meant to say here.

I'm also not convinced that intelligence bears unbounded payoff. There are limits to what any kind of intelligence can do, a superhuman AI couldn't come up with a faster than light propulsion or would disprove Gödel's incompleteness theorems.

Of course there are limits. That doesn't mean orders of magnitude better than a human isn't doable.

The point is, even if there are hitches and glitches that could stop a foom mid-way, they are like the size of golf courses compared to the size of the earth. No matter how many individual golf courses you propose for where a foom might be stopped, two thirds of the planet is still under water.

This is what LW reasoning refers to as "using arguments as soldiers": that is, treating the arguments themselves as the unit of merit, rather than the probability space covered by those arguments. I mean, are you seriously arguing that the only way to kick humankind's collective ass is by breaking the laws of math and physics? A being of modest intelligence could probably convince us all to do ourselves in, with or without tricky mind hacks or hypnosis!

The AI doesn't have to be that strong, because humans are so damn weak.

That it can simply invent it and then acquire it using advanced social engineering is pretty far-fetched in my opinion.

You would think so, but people apparently still fall for 419 scams. Human-level intelligence is more than sufficient to accomplish social engineering.

And what about taking over the Internet? It is not clear that the Internet would even be a sufficient substrate and that it could provide the necessary resources.

Today, presumably not. However, if you actually have a sufficiently-powered AI, then presumably, resources are available.

The thing is, foominess per se isn't even all that important to the overall need for FAI: you don't have to be that much smarter or faster than a human to be able to run rings around humanity. Historically, more than one human being has done a good job at taking over a chunk of the world, beginning with nothing but persuasive speeches!

Comment author: wedrifid 02 November 2010 01:11:45AM 3 points [-]

the probability of avoiding it by chance is virtually non-existent -- like hitting a golf ball in a random direction from a random spot on earth, and expecting it to score a hole in one.

I like the analogy. It may even fit when considering building a friendly AI - like hitting a golf ball deliberately and to the best of your ability from a randomly selected spot on the earth and trying to get a hole in one. Overwhelmingly difficult, perhaps even impossible given human capabilities but still worth dedicating all your effort to attempting!

Comment author: XiXiDu 02 November 2010 09:27:41AM *  3 points [-]

I don't know what you mean by "easy", or why it matters. The Foom argument is that, if you develop a sufficiently powerful AGI, it will foom, unless for some reason it doesn't want to.

What I meant is that you point out that a AGI will foom. Here your premises are that artificial general intelligence is feasible and that fooming is likely. Both premises are reasonable in my opinion. Yet you go one step further and use those arguments as a stepping stone for a further proposition. You claim that it is likely that the AGI (premise) will foom (premise) and that it will then run amog (conclusion). I do not accept the conclusion as given. I believe that it is already really hard to build AGI, or the seed for an AGI that is then able to rapidly self-improve itself. I believe that the level of insight and knowledge required will also allow one to constrain the AGI's sphere of action, its incentive not to fill the universe with as many paperclips as possible but merely a factory building.

But you don't get to simply say "I don't think that's likely", and call that evidence.

No you don't. But this argument runs in both directions. Note that I'm aware of the many stairways to hell by AGI here, the disjunctive arguments. I'm not saying they are not compelling enough to seriously consider them. I'm just trying to take a critical look here. There might be many pathways to safe AGI too, e.g. that it is really hard to build an AGI that cares at all. Hard enough to not get it to do much without first coming up with a rigorous mathematical definition of volition.

Without specific discussions of what "dangerous" and "impede AGI" mean in context, it's hard to separate that argument from an evidence-free heuristic.

Anything that might slow down the invention of true AGI even slightly. There are many risks ahead and without some superhuman mind we might not master them. So by anything you do that might slow down the development of AGI you have to take into account the possible increased danger from challenges an AGI could help to solve.

I don't understand why you think an AI couldn't use fuzziness or use brute force searches to accomplish the same things.

I believe it can, but also that this would mean that any AGI wouldn't be significantly faster than a human mind and really hard to self-improve. It is simply not known how effective the human brain is compared to the best possible general intelligence. Sheer bruteforce wouldn't make a difference then either, as humans could come up with such tools as quickly as the AGI.

This is what LW reasoning refers to as "using arguments as soldiers": that is, treating the arguments themselves as the unit of merit, rather than the probability space covered by those arguments.

If you do not compare probabilities then counter-arguments like the ones above will just outweigh your arguments. You've to show that some arguments are stronger than others.

You would think so, but people apparently still fall for 419 scams. Human-level intelligence is more than sufficient to accomplish social engineering.

Yes, but nobody is going to pull a chip-manufacture-factory out of thin air and hand it to the AGI. Without advanced nanotechnology the AGI will need the whole of humanity to help it develop new computational substrates.

Comment author: pjeby 02 November 2010 04:56:58PM 9 points [-]

You claim that it is likely that the AGI (premise) will foom (premise) and that it will then run amog (conclusion).

What I am actually claiming is that if such an AGI is developed by someone who does not sufficiently understand what the hell they are doing, then it's going to end up doing Bad Things.

Trivial example: the "neural net" that was supposedly taught to identify camouflaged tanks, and actually learned to recognize what time of day the pictures were taken.

This sort of mistake is the normal case for human programmers to make. The normal case. Not extraordinary, not unusual, just run-of-the-mill "d'oh" moments.

It's not that AI is malevolent, it's that humans are stupid. To claim that AI isn't dangerous, you basically have to prove that even the very smartest humans aren't routinely stupid.

So by anything you do that might slow down the development of AGI you have to take into account the possible increased danger from challenges an AGI could help to solve.

What I meant by "Without specific discussions" was, "since I haven't proposed any policy measures, and you haven't said what measures you object to, I don't see what there is to discuss." We are discussing the argument for why AGI development dangers are underrated, not what should be done about that fact.

It is simply not known how effective the human brain is compared to the best possible general intelligence.

Simple historical observation demonstrates that -- with very, very few exceptions -- progress is made by the people who aren't stuck in their perception of the way things are or are "supposed to be".

So, it's not necessary to know what the "best possible general intelligence" would be: even if human-scale is all you have, just fixing the bugs in the human brain would be more than enough to make something that runs rings around us.

Hell, just making something that doesn't use most of its reasoning capacity to argue for ideas it already has should be enough to outclass, say, 99.995% of the human race.

nobody is going to pull a chip-manufacture-factory out of thin air and hand it to the AGI.

What part of "people fall for 419 scams" don't you understand? (Hell, most 419 scams and phishing attacks suffer from being painfully obvious -- if they were conducted by someone doing a little research, they could be a lot better.)

People also fall for pyramid schemes, stock bubbles, and all sorts of exploitable economic foibles that could easily end up with an AI simply owning everything, or nearly everything, with nobody even the wiser.

Or, alternatively, the AI might fail at its attempts, and bring the world's economy down in the process.

If you do not compare probabilities then counter-arguments like the ones above will just outweigh your arguments. You've to show that some arguments are stronger than others.

Here's the argument: people are idiots. All people. Nearly all the time. Especially when it comes to computer programming.

The best human programmer -- the one who knows s/he's an idiot and does his/her best to work around the fact -- is still an idiot, and in possession of a brain that cannot be convinced to believe that it's really an idiot.(vs. all those other idiots out there), and thus still makes idiot mistakes.

The entire history of computer programming shows us that we think we can be 100% clear about what we mean/intend for a computer to do, and that we are wrong. Dead wrong. Horribly, horribly, unutterably wrong.

We are like, the very worst you can be at computer programming, while actually still doing it. We are just barely good enough to be dangerous.

That makes tinkering with making intelligent, self-motivating programs inherently dangerous, because when you tell that machine what you want it to do, you are still programming...

And you are still an idiot.

This is the bottom line argument for AI danger, and it isn't counterable until you can show me even ONE person whose computer programs never do anything that they didn't fully expect.and intend before they wrote it.

(It is also a supporting argument for why an AI needn't be all that smart to overrun humans -- it just has to not be as much of an idiot, in the ways that we are idiots, even if it's a total idiot in other ways we can't counter-exploit.)

Comment author: Perplexed 02 November 2010 05:23:15PM -1 points [-]

An outstanding piece of reasoning/rhetoric which deserves to be revised and relocated to top-level-postdom.

Comment author: XiXiDu 02 November 2010 05:51:00PM 3 points [-]

When programmers code faulty software then it usually fails to do its job. What you are suggesting is that humans succeed at creating the seed for an artificial intelligence with the incentive necessary to correct its own errors. It will know what constitutes an error based on some goal-oriented framework against which it can measure its effectiveness. Yet given this monumental achievement that includes the deliberate implementation of the urge to self-improve and the ability quantify its success, you cherry-pick the one possibility where somehow all this turns out to work except that the AI does not stop at a certain point but goes on to consume the universe? Why would it care to do so? Do you think it is that simple to tell it to improve itself yet hard to tell it when to stop? I believe it is vice versa, that it is really hard to get it to self-improve and very easy to constrain this urge.

Comment author: wedrifid 02 November 2010 06:11:46PM 2 points [-]

Do you think it is that simple to tell it to improve itself yet hard to tell it when to stop? I believe it is vice versa, that it is really hard to get it to self-improve and very easy to constrain this urge.

Your intuitions are not serving you well here. It may help to note that you don't have to tell an AI to self-improve at all. With very few exceptions giving any task to an AI will result in it self improving. That is, for an AI self improvement is an instrumental goal for nearly all terminal goals. The motivation to self improve in order to better serve its overarching purpose is such that it will find any possible loophole you leave if you try to 'forbid' the AI from self improving by any mechanism that isn't fundamental to the AI and robust under change.

Comment author: XiXiDu 02 November 2010 06:29:08PM *  1 point [-]

Whatever task you give an AI, you will have to provide explicit boundaries. For example, if you give an AI the task to produce paperclips most efficiently, then it shouldn't produce shoes. It will have to know very well what it is meant to do to be able to measure its efficiency against the realization of the given goal to be able to know what self-improvement means. If it doesn't know exactly what it should output it cannot judge its own capabilities and efficiency, it doesn't know what improvement implies.

How do you explain the discrepancy between implementing explicit design boundaries yet failing to implement scope boundaries?

Comment author: topynate 02 November 2010 06:21:28PM 3 points [-]

Just as a simple example, an AI could maximally satisfy a goal by changing human preferences so as to make us desire for it to satisfy that goal. This would be entirely consistent with constraints on not disobeying humans or their desires, while not at all in accordance with our current preferences or desired path of development.

Comment author: XiXiDu 02 November 2010 06:37:19PM 1 point [-]

Yes, but why would it do that? You seem to think that such unbounded creativity arises naturally in any given artificial general intelligence. What makes you think that rather than being impassive it would go on learning enough neuroscience to tweak human goals? If the argument is that AI's do all kinds of bad things because they do not care, why do they care to do a bad thing then rather than no-thing?

If you told the AI to make humans happy. It would first have to learn what humans are, what happiness means. Yet after learning all that you still expect it to not know that we don't like to be turned into broccoli? I don't think this is reasonable.

Comment author: sfb 02 November 2010 06:22:14PM 6 points [-]

When programmers code faulty software then it usually fails to do its job.

It often does it's job, but only in perfect conditions, or only once per restart, or with unwanted side effects, or while taking too long or too many resources or requiring too many permissions, or not keeping track that it isn't doing anything except it's job.

Buffer overflows for instance, are one of the bigger security failure causes, and are only possible because the software works well enough to be put into production while still having the fault present.

In fact, all production software that we see which has faults (a lot) works well enough to be put into production with those faults.

What you are suggesting is that humans succeed at creating the seed for an artificial intelligence with the incentive necessary to correct its own errors.

I think he's suggesting that humans will think we have succeeded at that, while not actually doing so (rigorously and without room for error).

Comment author: pjeby 02 November 2010 06:32:08PM 4 points [-]

you cherry-pick the one possibility where somehow all this turns out to work except that the AI does not stop at a certain point but goes on to consume the universe

It doesn't have to consume the universe. It doesn't even have to recursively self-improve, or even self-improve at all. Simple copying could be enough to say, wipe out every PC on the internet or accidentally crash the world economy.

(You know, things that human level intelligences can already do.)

IOW, to be dangerous, all it has to be able to affect humans, and be unpredictable -- either due to it being smart, or humans making dumb mistakes. That's all.

Comment author: JGWeissman 02 November 2010 06:35:22PM 0 points [-]

It will know what constitutes an error based on some goal-oriented framework against which it can measure its effectiveness.

If the error is in the goal-oriented framework, it could end up "correcting" itself to achieve unintended goals.

Comment author: Perplexed 02 November 2010 06:56:22PM 3 points [-]

Do you think it is that simple to tell it to improve itself yet hard to tell it when to stop? I believe it is vice versa, that it is really hard to get it to self-improve and very easy to constrain this urge.

I think it is important to realize that there are two diametrically opposed failure modes which SIAI's FAI research is supposed to prevent. One is the case that has been discussed so far - that an AI gets out of control. But there is another failure mode which some people here worry about. Which is that we stop short of FOOMing out of fear of the unknown (because FAI research is not yet complete) but that civilization then gets destroyed by some other existential risk that we might have circumvented with the assistance of a safe FOOMed AI.

As far as I know, SIAI is not asking Goertzel to stop working on AGI. It is merely claiming that its own work is more urgent than Goertzel's. FAI research works toward preventing both failure modes.

Comment author: timtyler 03 November 2010 07:48:02AM 2 points [-]

But there is another failure mode which some people here worry about. Which is that we stop short of FOOMing out of fear of the unknown (because FAI research is not yet complete) but that civilization then gets destroyed by some other existential risk that we might have circumvented with the assistance of a safe FOOMed AI.

I haven't seen much worry about that. Nor does it seem very likely - since research seems very unlikely to stop or slow down.

Comment author: JamesAndrix 02 November 2010 05:54:30AM 2 points [-]

I'm myself a bit suspicious if the argument for strong self-improvement is as compelling as it sounds though. Something you have to take into account is if it is possible to predict that a transcendence does leave your goals intact, e.g. can you be sure to still care about bananas after you went from chimphood to personhood.

Isn't that exactly the argument against non-proven AI values in the first place?

If you expect AI-chimp to be worried that AI-superchimp won't love bannanas , then you should be very worried about AI-chimp.

I don't get what you're saying about the paperclipper.

Comment author: XiXiDu 02 November 2010 09:04:08AM 1 point [-]

It is a reason not to transcend if you are not sure that you'll still be you afterwards, i.e. keep your goals and values. I just wanted to point out that the argument runs both directions. It is an argument for the fragility of values and therefore the dangers of fooming but also an argument for the difficulty that could be associated with radically transforming yourself.

Comment author: Jonathan_Graehl 01 November 2010 08:47:33PM 3 points [-]

any AI as smart as a human, can be expected to become MUCH smarter than human beings

(provided that the AI was originally built by humans, of course; if its design was too complicated for humans to arrive at, a slightly superhuman might be helpless as well)

Comment author: pjeby 02 November 2010 12:30:44AM 1 point [-]

(provided that the AI was originally built by humans, of course; if its design was too complicated for humans to arrive at, a slightly superhuman might be helpless as well)

Yes, that's rather the point. Assuming that you do get to human-level, though, you now have the potential for fooming, if only in speed.

Comment author: Bgoertzel 02 November 2010 01:45:37AM 18 points [-]

So, are you suggesting that Robin Hanson (who is on record as not buying the Scary Idea) -- the current owner of the Overcoming Bias blog, and Eli's former collaborator on that blog -- fails to buy the Scary Idea "due to cognitive biases that are hard to overcome." I find that a bit ironic.

Like Robin and Eli and perhaps yourself, I've read the heuristics and biases literature also. I'm not so naive as to make judgments about huge issues, that I think about for years of my life, based strongly on well-known cognitive biases.

It seems more plausible to me to assert that many folks who believe the Scary Idea, are having their judgment warped by plain old EMOTIONAL bias -- i.e. stuff like "fear of the unknown", and "the satisfying feeling of being part a self-congratulatory in-crowd that thinks it understands the world better than everyone else", and the well known "addictive chemical high of righteous indignation", etc.

Regarding your final paragraph: Is your take on the debate between Robin and Eli about "Foom" that all Robin was saying boils down to "la la la I can't hear you" ? If so I would suggest that maybe YOU are the one with the (metaphorical) hearing problem ;p ....

I think there's a strong argument that: "The truth value of "Once an AGI is at the level of a smart human computer scientist, hard takeoff is likely" is significantly above zero." No assertion stronger than that seems to me to be convincingly supported by any of the arguments made on Less Wrong or Overcoming Bias or any of Eli's prior writings.

Personally, I actually do strongly suspect that once an AGI reaches that level, a hard takeoff is extremely likely unless the AGI has been specifically inculcated with goal content working against this. But I don't claim to have a really compelling argument for this. I think we need a way better theory of AGI before we can frame such arguments compellingly. And I think that theory is going to emerge after we've experimented with some AGI systems that are fairly advanced, yet well below the "smart computer scientist" level.

Comment author: JamesAndrix 02 November 2010 05:42:01AM 1 point [-]

Regarding your final paragraph: Is your take on the debate between Robin and Eli about "Foom" that all Robin was saying boils down to "la la la I can't hear you" ?

I recall getting a distinct impression from Robin which I could caricature as "lalala you're biased with hero-epic story."

I also recall Eliezer asking for a probability breakdown, and I don't think Robin provided it.

Comment author: wedrifid 02 November 2010 07:36:30AM *  5 points [-]

I recall getting a distinct impression from Robin which I could caricature as "lalala you're biased with hero-epic story."

... and closely related: "I'm an Impressive Economist. If you don't just take my word for it you are arrogant."

In what I took to be an insightful comment by Eliezer in the aftermath of the debate Eliezer noted that he and Robin seemed to have fundamental disagreement about what should be taken as good evidence. This lead into posts about 'outside view', 'superficial similarities' and 'reference class tennis'. (And conceivably had something to do with priming the thoughts behind 'status and stupidity' although I would never presume that was primarily or significantly directed at Robin.)

Comment author: wedrifid 02 November 2010 07:22:04AM *  9 points [-]

Regarding your final paragraph: Is your take on the debate between Robin and Eli about "Foom" that all Robin was saying boils down to "la la la I can't hear you" ?

Good summary. Although I would have gone with "la la la la If you're right then most of expertise is irrelevant. Must protect assumptions of free competition. Respect my authority!"

What I found most persuasive about that debate was Robin's arguments - and their complete lack of merit. The absence of evidence is evidence of absence when there is a motivated competent debater with an incentive to provide good arguments.

Comment author: pjeby 02 November 2010 05:27:41PM *  8 points [-]

So, are you suggesting that Robin Hanson (who is on record as not buying the Scary Idea) -- the current owner of the Overcoming Bias blog, and Eli's former collaborator on that blog -- fails to buy the Scary Idea "due to cognitive biases that are hard to overcome." I find that a bit ironic

Welcome to humanity. ;-) I enjoy Hanson's writing, but AFAICT, he's not a Bayesian reasoner.

Actually: I used to enjoy his writing more, before I grokked Bayesian reasoning myself. Afterward, too much of what he posts strikes me as really badly reasoned, even when I basically agree with his opinion!

I similarly found Seth Roberts' blog much less compelling than I did before (again, despite often sharing similar opinions), so it's not just him that I find to be reasoning less well, post-Bayes.

(When I first joined LW, I saw posts that were disparaging of Seth Roberts, and I didn't get what they were talking about, until after I understood what "privileging the hypothesis" really means, among other LW-isms.)

I'm not so naive as to make judgments about huge issues, that I think about for years of my life, based strongly on well-known cognitive biases.

See, that's a perfect example of a "la la la I can't hear you" argument. You're essentially claiming that you're not a human being -- an extraordinary claim, requiring extraordinary proof.

Simply knowing about biases does very nearly zero for your ability to overcome them, or to spot them in yourself (vs. spotting them in others, where it's easy to do all day long.)

It seems more plausible to me to assert that many folks who believe the Scary Idea, are having their judgment warped by plain old EMOTIONAL bias -- i.e. stuff like "fear of the unknown", and "the satisfying feeling of being part a self-congratulatory in-crowd that thinks it understands the world better than everyone else", and the well known "addictive chemical high of righteous indignation", etc.

Since you said "many", I'll say that I agree with you that that is possible. In principle, it could be possible for me as well, but...

To be clear on my own position: I am a FAI skeptic, in the sense that I have a great many doubts about its feasibility -- too many to present or argue here. All I'm saying in this discussion is that to believe AI is dangerous, one only need to believe that humans are terminally stupid, and there is more than ample evidence for that proposition. ;-)

Also, more relevant to the issue of emotional bias: I don't primarily identify as an LW-ite; in fact I think that a substantial portion of the LW community has its head up its ass in overvaluing epistemic (vs. instrumental) rationality, and that many people here are emulating a level of reasoning they don't personally comprehend... and before I understood the reasoning myself, I thought the entire thing was a cult of personality, and wondered why everybody was making such a religious-sounding fuss over a minor bit of mathematics used for spam filtering. ;-)

Is your take on the debate between Robin and Eli about "Foom" that all Robin was saying boils down to "la la la I can't hear you" ?

My take is that before the debate, I was wary of AI dangers, but skeptical of fooming. Afterward, I was convinced fooming was near inevitable, given the ability to create a decent AI using a reasonably small amount of computing resources.

And a big part of that convincing was that Robin never seemed to engage with any of Eliezer's arguments, and instead either attacked Eliezer or said, "but look, other things happen this other way".

It seems to me that it'd be hard to do a worse job of convincing people of the anti-foom position, without being an idiot or a troll.

That is, AFAICT, Robin argued the way a lawyer argues when they know the client is guilty: pounding on the facts when the law is against them, pounding on the law when the facts are against them, and pounding on the table when the facts and law are both against.

I think there's a strong argument that: "The truth value of "Once an AGI is at the level of a smart human computer scientist, hard takeoff is likely" is significantly above zero."

Yep.

No assertion stronger than that seems to me to be convincingly supported by any of the arguments made on Less Wrong or Overcoming Bias or any of Eli's prior writings.

I'm curious what stronger assertion you think is necessary. I would personally add, "Humans are bad at programming, no nontrivial program is bug-free, and an AI is a nontrivial program", but I don't think there's a lack of evidence for any of these propositions. ;-)

[Edited to add the "given" qualification on "nearly inevitable", as that's been a background assumption I may not have made clear in my position on this thread.]

Comment author: Vladimir_Nesov 02 November 2010 05:38:53PM *  7 points [-]

I enjoy Hanson's writing, but AFAICT, he's not a Bayesian reasoner.

I don't believe it's a meaningful property (as used in this context), and you should do well to taboo it (possibly, to convince me it's actually meaningful).

Comment author: pjeby 02 November 2010 06:25:00PM 4 points [-]

I don't believe it's a meaningful property (as used in this context), and you should do well to taboo it

True enough; it would be more precise to say that he argues positions based on evidence which can also support other positions, and therefore isn't convincing evidence to a Bayesian.

Comment author: Vladimir_Nesov 02 November 2010 06:37:55PM 4 points [-]

it would be more precise to say that he argues positions based on evidence which can also support other positions, and therefore isn't convincing evidence to a Bayesian.

What do you mean? Evidence can't support both sides of an argument, so how can one inappropriately use such impossible evidence?

Comment author: pjeby 02 November 2010 06:46:13PM 4 points [-]

Evidence can't support both sides of an argument

What do you mean, "both"?

Comment author: wedrifid 02 November 2010 06:48:49PM 0 points [-]

What do you mean? Evidence can't support both sides of an argument, so how can one inappropriately use such impossible evidence?

It would be a mistake assume that PJ was limiting his evaluation to positions selected from one of those 'both sides' of a clear dichotomy. Particularly since PJ has just been emphasizing the relevance of 'privileging the hypothesis' to bayesian reasoning and also said 'other positions' plural. This being the case no 'impossible evidence' is involved.

Comment author: Vladimir_Nesov 02 November 2010 06:54:37PM *  1 point [-]

I see. But in that case, there is no problem with use of such evidence.

Comment author: sfb 02 November 2010 06:03:52PM 4 points [-]

PJ, I'd love to drag you off topic slightly and ask you about this:

before I understood the reasoning myself, I thought the entire thing was a cult of personality, and wondered why everybody was making such a religious-sounding fuss over a minor bit of mathematics used for spam filtering. ;-)

What is it that you now understand, that you didn't before?

Comment author: pjeby 02 November 2010 06:27:00PM 2 points [-]

What is it that you now understand, that you didn't before?

That is annoyingly difficult to describe. Of central importance, I think, is the notion of privileging the hypothesis, and what that really means. Why what we naively consider "evidence" for a position, really isn't.

ISTM that this is the core of grasping Bayesianism: not understanding what reasoning is, so much as understanding why what we all naively think is reasoning and evidence, usually isn't.

Comment author: sfb 02 November 2010 06:56:24PM 2 points [-]

That hasn't really helped... would you try again?

(What does privileging the hypothesis really mean? and why is reasoning and evidence usually ... not?)

Comment author: wedrifid 02 November 2010 07:05:51PM 2 points [-]

What does privileging the hypothesis really mean?

Have you come across the post by that name? Without reading that it may be hard to reverse engineer the meaning from the jargon.

The intro gives a solid intuitive description:

Suppose that the police of Largeville, a town with a million inhabitants, are investigating a murder in which there are few or no clues - the victim was stabbed to death in an alley, and there are no fingerprints and no witnesses.

Then, one of the detectives says, "Well... we have no idea who did it... no particular evidence singling out any of the million people in this city... but let's consider the hypothesis that this murder was committed by Mortimer Q. Snodgrass, who lives at 128 Ordinary Ln. It could have been him, after all."

That is privileging the hypothesis. When you start looking for evidence and taking an idea seriously when you have no good reason to consider it instead of countless others that are just as likely.

Comment author: sfb 02 November 2010 07:23:29PM *  3 points [-]

I have come across that post, and the story of the murder investigation, and I have an understanding of what the term means.

The obvious answer to the murder quote is that you look harder for evidence around the crimescene, and go where the evidence leads, and there only. The more realistic answer is that you look for recent similar murders, for people who had a grudge against the dead person, for criminals known to commit murder in that city... and use those to progress the investigation because those are useful places to start.

I'm wondering what pjeby has realised, which turns this naive yet straightforward understanding into wrongthought worth commenting on.

If evidence is not facts which reveal some result-options to be more likely true and others less likely true, then what is it?

Comment author: timtyler 02 November 2010 10:34:57PM *  4 points [-]

"Simply knowing about biases does very nearly zero for your ability to overcome them, or to spot them in yourself (vs. spotting them in others, where it's easy to do all day long.)"

I looked briefly at the evidence for that. Most of it seemed to be from the so-called "self-serving bias" - which looks like an adaptive signalling system to me - and so is not really much of a "bias" at all.

People are unlikely to change existing adaptive behaviour just because someone points it out and says it is a form of "bias". The more obvious thing to do is to conclude is that they don't know what they are talking about - or that they are trying to manipulate you.

Comment author: jmmcd 02 November 2010 02:27:24AM 0 points [-]

dumb brute force is already "smarter" than human beings in any narrow domain (see Deep Blue, evolutionary algorithms for antenna design, Emily Howell, etc.

I'm a fan of chess, evolutionary algorithms, and music, and the Emily Howell example is the one that sticks out like a sore thumb here. Music is not narrow and Emily Howell is not comparable to a typical human musician.

Comment author: pjeby 02 November 2010 06:14:54PM 2 points [-]

Music is not narrow and Emily Howell is not comparable to a typical human musician.

The point is that it (and its predecessor Emmy) are special-purpose "idiot savants", like the other two examples. That it is not a human musician is beside the point: the point is that humans can make idiot-savant programs suitable for solving any sufficiently-specified problem, which means a human-level AI programmer can do the same.

And although real humans spent many years on some of these narrow-domain tools, an AI programmer might be able to execute those years in minutes.

Comment author: jmmcd 02 November 2010 07:48:46PM 2 points [-]

special-purpose "idiot savants", like the other two examples.

No, it's quite different from the other two examples. Deep Blue beat the world champion. The evolutionary computation-designed antenna was better than its human-designed competitors.

dumb brute force is already "smarter" than human beings in any narrow domain

To be precise, what sufficiently-specified compositional problem do you think Emily Howell solves better than humans? I say "compositional" to reassure you that I'm not going to move the goalposts by requiring "real emotion" or human-style performance gestures or anything like that.

Comment author: pjeby 03 November 2010 04:04:57AM 1 point [-]

To be precise, what sufficiently-specified compositional problem do you think Emily Howell solves better than humans?

If I understand correctly, the answer would be "making the music its author/co-composer wanted it to make".

(In retrospect, I probably should have said "Emmy" -- i.e., Emily's predecessor that could write classical pieces in the style of other composers.)

Comment author: jmmcd 03 November 2010 10:15:30PM 1 point [-]

To make that claim, we'd have to have one or more humans who sat down with David Cope and tried to make the music that he wanted, and failed. I don't think David Cope himself counts, because he has written music "by hand" also, and I don't think he regards it as a failure.

Re EMI/Emmy, it's clearer: the pieces it produced in the style of (say) Beethoven are not better than would be written by a typical human composer attempting the same task.

Now would be a good time for me to acknowledge/recall that my disagreement on this doesn't take away from the original point -- computers are better than humans on many narrow domains.

Comment author: MatthewB 02 November 2010 06:11:30AM *  9 points [-]

From Ben Goertzel,

And I think that theory is going to emerge after we've experimented with some AGI systems that are fairly advanced, yet well below the "smart computer scientist" level.

At the second Singularity Summit, I heard this same sentiment from Ben, Robin Hanson, and from Rodney Brooks, and from Cynthia Breazeal (at the Third Singularity Summit), and from Ron Arkin (at the "Human Being in an Inhuman Age" Conference at Bard College on Oct 22nd ¹), and from almost every professor I have had (or will have for the next two years).

It was a combination of Ben, Robin and several professors at Berkeley and UCSD which led me to the conclusion that we probably won't know how dangerous an AGI (CGI - Constructed General Intelligence... Seems to be a term I have heard used by more than one person in the last year instead of AI/AGI. They prefer it to AI, as the word Artificial seems to imply that the intelligence is not real, and the word Constructed is far more accurate) is until we have put a lot more time into building AI (or CI) systems that will reveal more about the problems they attempt to address.

Sort of like how the Wright Brothers didn't really learn how they needed to approach building an airplane until they began to build airplanes. The final Wright Flyer didn't just leap out of a box. It is not likely that an AI will just leap out of a box either (whether it is being built at a huge Corporate or University lab, or in someone's home lab).

Also, it is possible that AI may come in the form of a sub-symbolic system which is so opaque that even it won't be able to easily tell what can or cannot be optimized.

Ron Arkin (From Georgia Tech) discussed this briefly at the conference at Bard College I mentioned.

MB

¹ I should really write up something about that conference here. I was shocked at how many highly educated people so completely missed the point, and became caught up in something that makes The Scary Idea seem positively benign in comparison.

Comment author: XiXiDu 01 November 2010 03:25:02PM *  20 points [-]

To be more precise. You can't tell concerned AI researchers to read through hundreds of posts of marginal importance. You have to have some brochure for experts and educated laymen to be able to read up on a summary of the big picture that includes precise and compelling methodologies that they can follow through to come up with their own estimations of the likelihood of existential risks posed by superhuman artificial general intelligence. If the decision procedure gives them a different probability due to a differing prior and values, then you can tell them to read up on further material to be able to update their prior probability and values accordingly.

Comment author: Kutta 01 November 2010 03:35:29PM *  3 points [-]

I'm content with your answer, then. I would personally welcome an overhaul to the presentation of AI material too. Still I think that Eliezer's FAI views are a lot more structured, comprehensive and accessible than the impression you give in your relevant posts.

Comment author: jimrandomh 01 November 2010 03:22:31PM 6 points [-]

If you want probabilities for these things to be backed up by mathematics, you're going to be disappointed, because there aren't any. The best probabilities - or rather, the only probabilities we have here, were produced using human intuition. You can break down the possibilities into small pieces, generate probabilities for the pieces, and get an overall probability that way, but at the base of the calculations you just have order-of-magnitude estimates. You can't provide formal, strongly defensible probabilities for the sub-events, because there just isn't any data - and there won't be any data until after the danger of AGI has destroyed us, or passed without destroying us. And that, I think, is the reason why SIAI doesn't provide any numbers: since they'd only be order of magnitude estimates, they'd give the people who had already made up their minds something to attack.

Comment author: XiXiDu 01 November 2010 03:42:23PM *  13 points [-]

I'm not asking for defensible probabilities that would withstand academic peer review. I'm asking for decision procedures including formulas with variables that allow you to provide your own intuitive values to eventually calculate your own probabilities. I want the SIAI to provide a framework that gives a concise summary of the risks in question and a comparison with other existential risks. I want people to be able to carry out results analysis and distinguish risks posed by artificial general intelligence from other risks like global warming or grey goo.

There aren't any numbers for a lot of other existential risks either. But one is still able to differentiate between those risks and that from unfriendly AI based on logical consequences of other established premises like the Church–Turing–Deutsch principle. Should we be equally concerned with occultists trying to summon world-changing supernatural powers?

Comment author: David_Gerard 01 November 2010 05:45:20PM *  10 points [-]

+1

Unfortunately, this is a common conversational pattern.

Q. You have given your estimate of the probability of FAI/cryonics/nanobots/FTL/antigravity. In support of this number, you have here listed probabilities for supporting components, with no working shown. These appear to include numbers not only for technologies we have no empirical knowledge of, but particular new scientific insights that have yet to occur. It looks very like you have pulled the numbers out of thin air. How did you derive these numbers?

A. Bayesian probability calculations.

Q. Could you please show me your working? At least a reasonable chunk of the Bayesian network you derived this from? C'mon, give me something to work with here.

A. (tumbleweeds)

Q. I remain somehow unconvinced.

If you pull a number out of thin air and run it through a formula, the result is still a number pulled out of thin air.

If you want people to believe something, you have to bother convincing them.

Comment author: Emile 01 November 2010 03:38:53PM *  7 points [-]

This might be an opportunity to use one of those Debate Tools, see if one of them can be useful for mapping the disagreement.

I would like to have a short summary of where various people stand on the various issues.

The people:

  • Eliezer

  • Ben

  • Robin Hanson

  • Nick Bostrom

  • Ray Kurzweil ?

  • Other academic AGI types?

  • Other vocal people on the net like Tim Tyler ?

The issues:

  • How likely is a human-level AI to go FOOM?

  • How likely is an AGI developed without "friendliness theory" to have values incompatible with those of humans?

  • How easy is it to make an AGI (really frickin' hard, or really really really frickin' hard?)?

  • How likely is it that Ben Goerzel's "toddler AGI" would succeed, if he gets funding etc.?

  • How likely is it that Ben Goerzel's "toddler AGI" would be dangerous, if he succeeded?

  • How likely is it that some group will develop an AGI before 2050? (Or more generally, estimated timelines of AGI)

Comment author: Kutta 01 November 2010 03:53:25PM 5 points [-]

Add Nick Bostrom to the list.

Also, what is exactly Bostrom's take on AI? OP says Bostrom disagrees with Eliezer. Could someone provide a link or reference to that? I have read most of Bostrom's papers some time ago and at the moment I can't recall any such disagreement.

Comment author: CarlShulman 01 November 2010 04:32:53PM *  4 points [-]

I think Nick was near Anders with an x-risk of 20% conditional on AI development by 2100, and near 50% for AI by 2100. So the most likely known x-risk, although unknown x-risks get a big chunk of his probability mass.

Comment author: Perplexed 01 November 2010 05:08:04PM *  1 point [-]

If we are constructing a survey of AI-singularity thinking here, I would like to know more about the opinions of Hugo de Garis. And what Bill Joy is thinking these days?

If we are trying to estimate probabilities and effect multipliers, I would like to consider the following question: Consider the projected trajectory of human technological progress without AGI assistance. For example: controlled fusion by 2140, human lifespan doubles by 2200, self-sustaining human presence on asteroids and/or Jovian satelites by 2260, etc. How much would that rate of progress be speeded if we had the assistance of AGI intelligence with 10x human speed and memory capacity? 100x? 1000x?

I conjecture that these speed-ups would be much less than people here seem to expect, and that the speed-up difference between 100x and 100,000x would be small. Intelligence may be much less important than many people think.

Comment author: timtyler 01 November 2010 08:44:08PM 2 points [-]

A recent update from Hugo here. He has retired - but says he has one more book on machine intelligence to go.

Comment author: Perplexed 01 November 2010 11:24:52PM 0 points [-]

Thx. From that interview:

Interviewer: So what's your take on Ben Goertzel's Cosmism, as expressed in "A Cosmist Manifesto"?

de Garis: Ben and I have essentially the same vision, i.e. that it’s the destiny of humanity to serve as the stepping-stone towards the creation of artilects. Where we differ is on the political front. I don’t share his optimism that the rise of the artilects will be peaceful. I think it will be extremely violent — an artilect war, killing billions of people.

Hmmm. I'm afraid I don't share Goertzel's optimism either. But then I don't buy into that "destiny" stuff, either. We don't have to destroy ourselves and the planet in this way. It is definitely not impossible, but super-human AGI is also not inevitable.

I'd be curious to hear from EY, and the rest of the "anti-death" brigade here, what they think of de Garis's prognosis and whether and how they think an "artilect war" can be avoided.

Comment author: ata 02 November 2010 12:57:54AM 5 points [-]

I'd be curious to hear from EY, and the rest of the "anti-death" brigade here, what they think of de Garis's prognosis and whether and how they think an "artilect war" can be avoided.

I'm not sure that's where the burden of proof should fall. Has de Garis justified his claim? It sounds more like storytelling than inferential forecasting to me.

Comment author: Perplexed 02 November 2010 02:01:30AM 1 point [-]

I haven't read his book, etc., but I suspect that "storytelling" might be a reasonable characterization. On the other hand, my "I'd be curious" was hardly an attempt to create a burden of proof.

I do personally believe that convincing mankind that an FAI singularity is desirable will be a difficult task, and that many sane individuals might consider a unilateral and secret decision to FOOM as a casus belli. What would you do as Israeli PM if you received intelligence that an Iranian AI project would likely go FOOM sometime within the next two months?

Comment author: MichaelVassar 03 November 2010 10:11:48PM 3 points [-]

I really like your comments and wish you would make some top level posts and also contact me online. Could you please do so?

Comment author: ata 03 November 2010 11:11:22PM 0 points [-]

Where shall I contact you?

Comment author: timtyler 02 November 2010 08:43:52AM *  0 points [-]

It's just silly. Luddites have never had much power - and aren't usually very war like.

Instead, we will see expanded environmental and green movements, more anti-GM activism - demands to tax the techno-rich more - and so on.

Degaris was just doing much the same thing that SIAI is doing now - making a song-and-dance about THE END OF THE WORLD - in order to attract attention to himself - and so attract funding - so he could afford to get on with building his machines.

Comment author: timtyler 02 November 2010 10:13:52PM *  1 point [-]

Consider the projected trajectory of human technological progress without AGI assistance. For example: controlled fusion by 2140, human lifespan doubles by 2200, self-sustaining human presence on asteroids and/or Jovian satelites by 2260, etc. How much would that rate of progress be speeded if we had the assistance of AGI intelligence with 10x human speed and memory capacity? 100x? 1000x?

I don't think you can say. Different things will accelerate at different rates. For example, a dog won't build a moon rocket in a million years - but if you make it 10 times smarter, it might do that pretty quickly.

Comment author: jimrandomh 01 November 2010 03:40:39PM *  11 points [-]

The claim that AIs will foom, basically, reduces to the claim that the difficulty of making AGI is front-loaded: that there's a hump to get over, that we aren't over it yet, and that once it's passed things will get much easier. From an outside view, this makes sense; we don't yet have a working prototype of general intelligence, and the history of invention in general indicates that the first prototype is a major landmark after which the pace of development speeds up dramatically.

But this is a case where the inside and outside views disagree. We all know that AGI is hard, but the people actually working on it get to see the challenges up close. And from that perspective, it's hard to accept that it will suddenly become much easier once we have a prototype - both because the challenges seem so daunting, the possible breakthroughs are hard to visualize, and on some level, if AI suddenly became easy it would trivialize the challenges that researchers are facing now. So the AGI researchers imagine an AI-Manhattan Project, with resources to match the challenges as they see them, rather than an AI-Kitty Hawk, with a few guys in a basement who are lucky enough to stumble on the final necessary insight.

Since a Manhattan Project-style AI would have lots of resources to spend on ensuring safety, the safety issues don't seem like a big deal. But if the first AGI were made by some guys in a basement, instead, then they won't have those resources; and from that perspective, pushing hard for safety measures is important.

Comment author: JamesAndrix 01 November 2010 10:53:59PM 6 points [-]

Except in this case if 'prototype' means genius-human-level AI, then it's reasonable to assume that even if the further challenges remain daunting, it will be economical to put a lot more effort into them, because researchers will be cheap.

If airplanes were as much better at designing airplanes as they are at flying, Kitty Hawk would have been different.

Comment author: rwallace 02 November 2010 05:45:51PM 1 point [-]

The claim that AIs will foom, basically, reduces to the claim that the difficulty of making AGI is front-loaded

Yes.

the history of invention in general indicates that the first prototype is a major landmark after which the pace of development speeds up dramatically.

This is not actually true. The history of invention in general indicates that the first prototype accomplishes little, and a great deal of subsequent work needs to be done - even in the case of inventions like machine tools and computers that are used for creating subsequent generations of themselves.

Comment author: CarlShulman 04 November 2010 07:58:02PM 2 points [-]

Yes, this is right. Prototypes often precede widespread deployment and impact of a technology by decades until various supporting technologies and incremental improvements make them worth their costs.

Comment author: steven0461 04 November 2010 08:15:35PM 2 points [-]

The claim that AIs will foom, basically, reduces to the claim that the difficulty of making AGI is front-loaded

Or that the effective effort put into AI research (e.g. by AIs) is sufficiently back-loaded.

Comment author: multifoliaterose 01 November 2010 04:08:20PM *  6 points [-]

Great post.

If you haven't seen SIAI's new overview you might find it relevant. I'm quite favorably impressed by it.

Comment author: XiXiDu 01 November 2010 04:44:38PM *  5 points [-]

Thanks. I actually linked to that paper in the OP. As I wrote, that an organisation like the SIAI is necessary and should be supported is not being challenged. But what that paper accomplishes is merely giving a very basic introduction to someone who might have never thought about risks posed by AI. What I actually had in mind writing the OP is that the SIAI addresses people like Ben Goertzel who are irrespective of the currently available material skeptic about the risks from working on AGI and who are unsure of what the SIAI actually wants them to do or not to do and why. Further I would like if the SIAI provided educated outsiders with a summary of how people that believe into the importance of risks associated with AGI arrived at this conclusion, especially in comparison to other existential risks and challenges.

What I seek is a centralized code of practice that incorporates the basic assumptions and a way to roughly asses their likelihood in comparison to other existential risks and challenges by the use of probability. See for example this SIAI page. Bayes sits in there alone and doomed. Why is there no way for people to formally derive their own probability estimates with their own values? To put it bluntly, it looks like you have to put any estimation out of your ass. The SIAI has to set itself apart from works of science fiction and actually provide some formal analysis of what we know, what conclusions can be drawn and how they relate to other problems. The first question most people will ask is why to worry about AGI when there are challenges like climate change. There needs to be a risks benefits analysis that shows why AGI is more important and a way to reassess the results yourself by following a provided decision procedure.

Comment author: multifoliaterose 01 November 2010 04:51:47PM 2 points [-]

Yes, I strongly endorse what you were saying in your top level posting and agree that the new overview is by no means sufficient, I was just remarking that the new overview is at least a step in the right direction. Didn't notice that you had linked it in the top level post.

Comment author: [deleted] 01 November 2010 05:03:53PM 0 points [-]

AFAICT the main value in addressing such concerns in detail consists would lie in convincing AGI researchers to change their course of action. Do you think this would actually occur?

Comment author: Jonathan_Graehl 01 November 2010 08:44:33PM 0 points [-]

It depends on how convincing the argument is.

I think 1 in 5 would change if they heard a obviously correct argument.

Comment author: [deleted] 02 November 2010 05:50:02AM *  0 points [-]

Do you think this would actually occur?

To answer this question it seems like it would be useful to understand the motives behind those researching AGI. I don't know much about this. Maybe those who have interacted with these researchers (or the researchers themselves) can shed some light?

Comment author: CarlShulman 01 November 2010 05:20:26PM 10 points [-]

I have a lot on my plate right now, but I'll try to write up my own motivating Fermi calculations if I get the chance to do so soon.

Comment author: utilitymonster 01 November 2010 09:07:30PM 2 points [-]

Would like to see it.

Comment author: Jordan 01 November 2010 05:35:04PM 7 points [-]

I would additionally like to see addressed:

  • What is the time estimate for FAI and AGI?
  • What is the probability that is FAI is possible times the probability that FAI can be achieved before AGI?
  • Other paths to safe super intelligence (IA, WBE, AI-in-box, etc) may be more dangerous. What are the odds? Are the odds better or worse than the odds that the FAI research program is successful?
Comment author: komponisto 01 November 2010 05:35:49PM 16 points [-]

Stop taking the numbers so damn seriously, and think in terms of subjective probability distributions [...], Michael Anissimov

I think it's worth giving the full quote:

Stop taking the numbers so damn seriously, and think in terms of subjective probability distributions, discard your mental associates between numbers and absolutes, and my choice to say a number, rather than a vague word that could be interpreted as a probability anyway, makes sense. Working on www.theuncertainfuture.com, one of the things I appreciated the most were experts with the intelligence to make probability estimates, which can be recorded, checked, and updated with evidence, rather than vague statements like “pretty likely”, which have to be converted into probability estimates for Bayesian updating anyway. Futurists, stick your neck out! Use probability estimates rather than facile absolutes or vague phrases that mean so little that you are essentially hedging yourself into meaninglessness anyway.

Total agreement from me, needless to say.

Comment author: steven0461 01 November 2010 09:20:32PM 8 points [-]

What I would like the SIAI to publish

Publish instead of doing what?

Comment author: Bgoertzel 02 November 2010 01:30:38AM 12 points [-]

I agree that a write-up of SIAI's argument for the Scary Idea, in the manner you describe, would be quite interesting to see.

However, I strongly suspect that when the argument is laid out formally, what we'll find is that

-- given our current knowledge about the pdf's of the premises in the argument, the pdf on the conclusion is verrrrrrry broad, i.e. we can't conclude hardly anything with much of any confidence ...

So, I think that the formalization will lead to the conclusion that

-- "we can NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly lead to bad consequences for humanity"

-- "we can also NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly NOT lead to bad consequences for humanity"

I.e., I strongly suspect the formalization

-- will NOT support the Scary Idea

-- will also not support complacency about AGI safety and AGI existential risk

I think the conclusion of the formalization exercise, if it's conducted, will basically be to reaffirm common sense, rather than to bolster extreme views like the Scary Idea....

-- Ben Goertzel

Comment author: MatthewB 02 November 2010 05:34:43AM 2 points [-]

I agree.

I doubt you would remember this, but we talked about this at the Meet and Greet at the Singularity Summit a few months ago (in addition to CBGBs and Punk Rock and Skaters).

James Hughes mentioned you as well at a Conference in NY where we discussed this very issue as well.

One thing that you mentioned at the Summit (well in conversation) was that The Scary Idea was tending to cause some paranoia among people who otherwise might be contributing more to the development of AI (of course, you also seemed pretty hostile to brain emulation too) as it tends to cause funding that could be going to AI to be slowed as a result.

Comment author: CarlShulman 04 November 2010 07:49:15PM *  5 points [-]

So, I think that the formalization will lead to the conclusion that "we can NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly lead to bad consequences for humanity" "we can also NOT confidently say, now, that: Building advanced AGI without a provably Friendly design will almost certainly NOT lead to bad consequences for humanity"

I agree with both those statements, but think the more relevant question would be:

"conditional on it turning out, to the enormous surprise of most everyone in AI, that this AGI design is actually very close to producing an 'artificial toddler', what is the sign of the expected effect on the probability of an OK outcome for the world, long-term and taking into account both benefits and risks?" .

Comment author: lukeprog 10 October 2012 08:10:28AM *  0 points [-]

[deleted]

Comment author: mindviews 02 November 2010 02:46:41AM 11 points [-]

For those of you who are interested, some of us folks from the SoCal LW meetups have started working on a project that seems related to this topic.

We're working on building a fault tree analysis of existential risks with a particular focus on producing a detailed analysis of uFAI. I have no idea if our work will at all resemble the decision procedure SIAI used to prioritize their uFAI research, but it should at least form a framework for the broader community to discuss the issue. Qualitatively you could use the work discuss the possible failure modes that would lead to a uFAI scenario and quantitatively you can could use the framework and your own supplied probabilities (or aggregated probabilities from the community, domain experts, etc.) to crunch the numbers and/or compare uFAI to other posited existential risks.

At the moment, I'd like to find out generally what anyone else thinks of this project. If you have suggestions, resources or pointers to similar/overlapping work you want to share, that would be great, too.

Comment author: PeerInfinity 02 November 2010 03:35:13PM *  4 points [-]

This project sounds really interesting and useful.

It sounds a lot like a project that I tried and failed to get started. Or at least like part of that project. Though my project is so vague and broad that pretty much anything involving graphs/trees related to x-risks would seem "kinda like part of the project I was working on"

Here's a link to another comment about that project

I would like to hear more about your project.

Comment author: MichaelVassar 03 November 2010 10:04:45PM 3 points [-]

It's my professional opinion, based on extensive experience and a developed psychological model of human rationality, that such a paper wouldn't be useful. That said, I'd be happy to have you attempt it. I think that your attempt to do so would work perfectly well for your and our purposes, at least if you are able to do the analysis honestly and update based on criticism that you could get in the comments of a LW blog post.

Comment author: XiXiDu 04 November 2010 10:11:58AM 4 points [-]

Thanks, but my current level of education is completely insufficient to accomplish such a feat to an extent that would be adequate. Maybe in a few years, but right now that is unrealistic.

Comment author: XiXiDu 04 November 2010 11:42:36AM *  2 points [-]

I added a footnote to the post:

  • Potential negative consequences [3] of slowing down research on artificial intelligence (a risks and benefits analysis).

(3) Could being overcautious be itself an existential risk that might significantly outweigh the risk(s) posed by the subject of caution? Suppose that most civilizations err on the side of caution. This might cause them to either evolve much slower so that the chance of a fatal natural disaster to occur before sufficient technology is developed to survive it, rises to 100%, or stops them from evolving at all for being unable to prove something being 100% safe before trying it and thus never taking the necessary steps to become less vulnerable to naturally existing existential risks. Further reading: Why safety is not safe

Comment author: NancyLebovitz 04 November 2010 11:46:45AM 0 points [-]

I was thinking about how the existential risks affect each other-- for example, a real world war might either destroy so much that high tech risks become less likely for a while, or lead to research which results in high tech disaster.

And we may get home build-a-virus kits before AI is developed, even if we aren't cautious about AI.

Comment author: XiXiDu 06 November 2010 08:03:01PM 2 points [-]

Interesting, this is why I included the Fermi paradox:

...so I must wonder: what big things future could go wrong where analogous smaller past things can’t go wrong? Many of you will say “unfriendly AI” but as Katja points out a powerful unfriendly AI that would make a visible mark on the universe can’t be part a future filter; we’d see the paperclips out there.

Comment author: XiXiDu 11 November 2010 10:11:55AM *  0 points [-]

SIA says AI is no big threat

Artificial Intelligence could explode in power and leave the direct control of humans in the next century or so. It may then move on to optimize the reachable universe to its goals. Some think this sequence of events likely.

If this occurred, it would constitute an instance of our star passing the entire Great Filter. If we should cause such an intelligence explosion then, we are the first civilization in roughly the past light cone to be in such a position. If anyone else had been in this position, our part of the universe would already be optimized, which it arguably doesn’t appear to be. This means that if there is a big (optimizing much of the reachable universe) AI explosion in our future, the entire strength of the Great Filter is in steps before us.

This means a big AI explosion is less likely after considering the strength of the Great Filter, and much less likely if one uses the Self Indication Assumption (SIA).

SIA implies that we are unlikely to give rise to an intelligence explosion for similar reasons, but probably much more strongly.

Comment author: timtyler 21 November 2010 01:06:46PM *  -1 points [-]

This summary seems fairly accurate:

In summary, if you begin with some uncertainty about whether we precede an AI explosion, then updating on the observed large total filter and accepting SIA should make you much less confident in that outcome.

The utility of an anthropic approach to this issue seems questionable, though. The great silence tells something - something rather depressing - it is true... but it is far from our only relevant source of information on the topic. We have an impressive mountain of other information to consider and update on.

To give but one example, we don't yet see any trace of independently-evolved micro-organisms on other planets. The less evidence for independent origins of life elsewhere there is, the more that suggests a substantial early filter - and the less need there is for a late one.

This is true - but because it does not suggest THE END OF THE WORLD - it is not so newsworthy. Selective reporting favours apocalyptic elements. Seeing only the evidence supporting one side of such stories seems likely to lead to people adopting a distorted world view, with inacurate estimates of the risks.