Furcas comments on Muehlhauser-Goertzel Dialogue, Part 2 - Less Wrong

9 Post author: lukeprog 05 May 2012 12:21AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (52)
Tags: singularity rationality

Comments (52)

You are viewing a single comment's thread.

Comment author: Furcas 05 May 2012 09:59:03AM *  5 points [-]

We still haven't gotten a decent reply to,

I remain confused about why an intelligent system will decide that a particular final goal it has been given is "stupid," and then change its final goals — especially given the convergent instrumental goal to preserve its final goals.

Unless you think that nonsense about being "out of harmony with the Cosmos" is a decent reply.

Comment author: timtyler 05 May 2012 03:10:04PM *  2 points [-]

What Ben originally said was:

if you tried to build a superintelligence with this goal and connected it to the real world, it would very likely get its initial goal subverted and wind up pursuing some different, less idiotic goal.

One possibility is that it gets shut down by its makers - who then go on to build a more useful machine. Another possibility is that it gets shut down by the government. Silly goals won't attract funding or support, and such projects are likely to be overtaken by better-organised ones that provide useful services.

I think we need a "taking paperclipper scenario seriously" FAIL category.

Comment author: jsalvatier 05 May 2012 03:32:40PM -1 points [-]

I was confused about this too, and this helped me make a bit more sense of that.

Powered by Reddit Powered by Reddit