TheAncientGeek comments on Debunking Fallacies in the Theory of AI Motivation - Less Wrong

8 Post author: Richard_Loosemore 05 May 2015 02:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (343)

You are viewing a single comment's thread. Show more comments above.

Comment author: Viliam 05 May 2015 10:35:14AM *  16 points [-]

his conviction that every AI, no matter how well it is designed, will turn into a gobbling psychopath is just one of many doomsday predictions being popularized in certain sections of the AI community

What is your probability estimate that an AI would be a psychopath, if we generalize the meaning of "psychopath" beyond individuals from homo sapiens species as "someone who does not possess precisely tuned human empathy"?

(Hint: All computer systems produced until today are psychopaths by this definition.)

[is an AI that is superintelligent enough to be unstoppable] and [believes that benevolence toward humanity might involve forcing human beings to do something violently against their will.]

The idea of the second statement is that "benevolence" (as defined by the AI code) is not necessarily the same thing as benevolence (as humans understand it). Thus the AI may believe -- correctly! -- that forcing human beings to do something against their will is "benevolent".

The AI is superintelligent, but its authors are not. If the authors write a code to "maximize benevolence as defined by the predicate B001", the AI will use its superintelligence to maximize B001. Even if the AI would realize that B001 is not what humans mean by benevolence, it would not care, because it is programmed to maximize B001.

Instead you are suggesting that the superintelligent AI programmed to maximize B001 will look at humans and say "oh, those idiots programmed me to maximize B001 when in fact they would prefer me to maximize B002... so I am modifying myself to maximize B002 instead of B001". Why exactly would a machine programmed to maximize B001 do that?

Comment author: TheAncientGeek 06 May 2015 10:56:05AM *  5 points [-]

If we define a psychopath as an entity with human like egoistic drives, but no human like empathy, it turns out that no preent computer systems are psychopaths.