Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It)

Paul Crowley

[...] SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.

[...] Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)

So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.

[...] If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.

The line of argument makes sense, if you accept the premises.

But, I don't.

Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It), October 29 2010. Thanks to XiXiDu for the pointer.

[...] SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.

So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.

Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It), October 29 2010. Thanks to XiXiDu for the pointer.

the Scary Idea is privileging a hypothesis

No. Expecting a superintelligence to optimize for our specific values would be privileging a hypothesis. The "Scary Idea" is saying that most likely something else will happen.

Expecting a superintelligence to optimize for our specific values would be privileging a hypothesis. The "Scary Idea" is saying that most likely something else will happen.

That is not exactly what Goertzel meant by "Scary Idea". He wrote:

Roughly, the Scary Idea posits that: If I or anybody else actively trying to build advanced AGI succeeds, we're highly likely to cause an involuntary end to the human race.

It seems to me that there may be a lot of wiggle room in between failing to "optimize for our specific values" and... (read more)

1Vaniver16y

I may have to start only writing thousand-word replies, in the hopes that I can communicate more clearly in such a format. There are two aspects to the issue of how much work should be put into FAI as I understand it. The first I word like this- "the more thought we put into whether or not an AGI will be friendly, the more likely the AGI will be friendly." The second I word like this- "the more thought we put into making our AGI, the less likely our AGI will be the AGI." Both are wrapped up in the Scary Idea- the first part is it as normally stated, the second part is its unstated consequence. The value of believing the Scary Idea is the benefit of the first minus the cost of the second. My understanding is that we have no good estimation of the value of the first aspect or the second aspect. This isn't astronomy where we have a good idea of the number of asteroids out there and a pretty good idea of how they move through space. And so, to declare that the first aspect is stronger without evidence strikes me as related to privileging the hypothesis. (I should note that I expect, without evidence, the problem of FAI to be simpler than the problem of AGI, and thus don't think the Scary Idea has any policy implications besides "someone should work on FAI." The risk that AGI gets solved before FAI means more people should work on FAI, not that less people should work on AGI.)

42

Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It)

42

42

42

Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It)

42

42