Ultima comments on Perfectly Friendly AI - Less Wrong

7 Post author: Desrtopa 24 January 2011 07:03PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread. Show more comments above.

Comment author: Ultima 24 January 2011 07:52:46PM *  3 points [-]

I suspect that quite a few humans (including me) would be horrified by the actual implications of their own values being maximally realized.

How exactly could you be "horrified" about that unless you were comparing some of your own values being maximally realized with some of your other values not being maximally realized?

In other words, it doesn't make sense (doesn't even mean anything!) to say that you would be horrified (isn't that a bad thing?) to have your desires fulfilled (isn't that a good thing?), unless you're really just talking about some of your desires conflicting with some of your other desires.

Comment author: TheOtherDave 24 January 2011 08:09:22PM 4 points [-]

If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn't what I'm talking about).

My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.

Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that's all we ever do.

But even if I'm wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren't aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we'll experience that as emotionally unpleasant and challenging.

Comment author: Ultima 24 January 2011 08:23:32PM 1 point [-]

Solid answer, as far as I can see right now.

Comment author: David_Gerard 24 January 2011 11:48:53PM *  4 points [-]

Because humans' values are not a coherent, consistent set. We execute an evolutionarily-determined grab-bag of adaptations; there is no reason to assume this grab-bag adds up to a more coherent whole than "don't die out." (And even that's a teleological projection onto stuff just happening.)