Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

PhilGoetz comments on SIAI - An Examination - Less Wrong

143 Post author: BrandonReinhart 02 May 2011 07:08AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (203)

You are viewing a single comment's thread. Show more comments above.

Comment author: PhilGoetz 15 May 2011 04:57:12AM *  3 points [-]

An AI that is successfully "Friendly" poses an extistential risk of a kind that other AIs don't pose. The main risk from an unfriendly AI is that it will kill all humans. That isn't much of a risk; humans are on the way out in any case. Whereas the main risk from a "friendly" AI is that it will successfully impose a single set of values, defined by hairless monkeys, on the entire Universe until the end of time.

And, if you are afraid of unfriendly AI because you're afraid it will kill you - why do you think that a "Friendly" AI is less likely to kill you? An "unfriendly" AI is following goals that probably appear random to us. There are arguments that it will inevitably take resources away from humans, but these are just that - arguments. Whereas a "friendly" AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you're already on its list of people whose continued existence will be risky.

(Also, all these numbers seem to be pulled out of thin air.)

Comment author: nshepperd 15 May 2011 02:03:37PM 6 points [-]

I see no reason an AI with any other expansionist value system will not exhibit the exact same behaviour, except towards a different goal. There's nothing so special about human values (except that they're, y'know, good, but that's a different issue).

Comment author: Rain 15 May 2011 01:42:19PM *  5 points [-]

You're using a different definition of "friendly" than I am. An 80% chance SIAI's AI is Unfriendly already contains all of your "takes over but messes everything up in unpredictable ways" scenarios.

The numbers were exaggerated for effect, to show contrast and my thought process. It seems to me that you think the probabilities are reversed.

Comment author: timtyler 15 May 2011 02:32:53PM *  3 points [-]

And, if you are afraid of unfriendly AI because you're afraid it will kill you - why do you think that a "Friendly" AI is less likely to kill you?

One definition of the term explains:

The term "Friendly AI" refers to the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.

See the "non-human-harming" bit. Regarding:

If your name appears on this website, you're already on its list of people whose continued existence will be risky.

Yes, one of their PR problems is that they are implicitly threatening their rivals. In the case of Ben Goertzel some of the threats are appearing IRL. Let us hear the tale of how threats and nastiness will be avoided. No plan is not a good plan, in this particular case.

Comment author: TimFreeman 15 May 2011 06:22:01PM 2 points [-]

An AI that is successfully "Friendly" poses an extistential risk of a kind that other AIs don't pose. The main risk from an unfriendly AI is that it will kill all humans. That isn't much of a risk

What do you mean by existential risk, then? I thought things that killed all humans were, by definition, existential risks.

humans are on the way out in any case.

What, if anything, do you value that you expect to exist in the long term?

There are arguments that [an UFAI] will inevitably take resources away from humans, but these are just that - arguments.

Pretty compelling arguments, IMO. It's simple -- the vast majority of goals can be achieved more easily if one has more resources, and humans control resources, so an entity that is able to self-improve will tend to seize control of all the resources and therefore take control of those resources from the humans.

Do you have a counterargument, or something relevant to the issue that isn't just an argument?

Comment author: wedrifid 15 May 2011 03:40:50PM *  1 point [-]

AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you're already on its list of people whose continued existence will be risky.

Not much risk. Hunting down irrelevant blog commenters is a greater risk than leaving them be. There isn't much of a window during which any human is a slightest threat and during that window going around killing people is just going to enhance the risk to it.

Comment author: timtyler 15 May 2011 04:15:06PM *  2 points [-]

The window is presumably between now and when the winner is obvious - assuming we make it that far.

IMO, there's plenty of scope for paranoia in the interim. Looking at the logic so far some teams will reason that unless their chosen values get implemented, much of value is likely to be lost. They will then mulitiply that by a billion years and a billion planets - and conclude that their competitors might really matter.

Killing people might indeed backfire - but that still leaves plenty of scope for dirty play.

Comment author: wedrifid 15 May 2011 04:25:24PM 2 points [-]

The window is presumably between now and when the winner is obvious

No. Reread the context. This is the threat from "F"AI, not from designers. The window opens when someone clicks 'run'.

Comment author: timtyler 15 May 2011 05:54:30PM *  0 points [-]

Uh huh. So: world view difference. Corps and orgs will most likely go from 90% human to 90% machine through the well-known and gradual process of automation, gaining power as they go - and the threats from bad organisations are unlikely to be something that will appear suddenly at some point.