Konkvistador comments on Is friendly AI "trivial" if the AI cannot rewire human values? - Less Wrong

-5 Post author: Alerus 09 May 2012 05:48PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (57)

You are viewing a single comment's thread.

Comment author: [deleted] 09 May 2012 07:54:11PM *  7 points [-]

Finally, we will also assume that the AI does not possess the ability to manually rewire the human brain to change what a human values. In other words, the ability for the AI to manipulate another person's values is limited by what we as humans are capable of today. Given all this, is there any concern we should have about making this AI; would it succeed in being a friendly AI?

AAaiiiieee!

Can you please think about the emphasised phrase by yourself for 5 minutes (as measured by a physical clock)?

Comment author: faul_sname 10 May 2012 02:35:51AM 0 points [-]

That was approximately my reaction. Considering how easy it is to influence humans, it would be a nontrivial task to avoid unintentionally changing those values. I suspect that a significantly subhuman AI could change human values. One might argue that by increasing and redistributing wealth and improving medical care, our expert systems today are changing human value unintentionally. Google ads, which are automated and "learn" what the user likes, are specifically tailored to influence us, and do a remarkable job considering the relative simplicity of the algorithms used.

Comment author: Alerus 09 May 2012 09:00:55PM *  0 points [-]

What is wrong with the statement? The idea I'm trying to portray is that I as a person now, cannot go and forcefully rewire another person's values. The only ability I have to try an affect them is to be persuasive in argument or perhaps being deceptive about certain things to try and get them to a different position (e.g., consider the state of politics).

In contrast, one of the concerns for the future is that an AI may have the technological ability to more directly manipulate a person. So the question I'm asking is: is the future technology at the disposal of an AI the only reason it could behave "badly?" under such a utility function?

Also, please avoid such comments. I am interested in having this discussion, but alluding to finding something wrong in what I have posted and not saying what you think it is, is profoundly unhelpful and useless to discussion.

Comment author: drethelin 09 May 2012 09:23:15PM 4 points [-]

Consider that humans have modified human values to results as different as nazism and as jainism.

Comment author: DanArmak 09 May 2012 09:36:15PM 3 points [-]

Consider that every human who ever existed, was shaped purely by environment + genes.

Consider how much humans have achieved merely by controlling the environment: converting people to insane religions which they are willing to die and kill for, making torturers, "the banality of evil", etc. etc.

Now imagine what an entity could achieve with that plus 1) complete understanding of how the brain is shaped by the environment and/or 2) complete control of the environment (via VR, smart dust, whatever) for a human from age 0 onwards.

I think the conservative assumption is that any mind we would recognize as human, and many we wouldn't, could be produced by such an optimization process. You're not limiting your AI at all.