TheOtherDave comments on Perfectly Friendly AI - Less Wrong

7 Post author: Desrtopa 24 January 2011 07:03PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

You are viewing a single comment's thread. Show more comments above.

Comment author: TheOtherDave 24 January 2011 08:09:22PM 4 points [-]

If I am completely and consistently aware of what I actually value, then yes, my desires are equivalent to my values and it makes no sense to talk about satisfying one while challenging the other (modulo cases of values-in-tension, as you say, which isn't what I'm talking about).

My experience is that people are not completely and consistently aware of what they actually value, and it would astonish me if I turned out to be the fortunate exception.

Humans frequently treat instrumental goals as though they were terminal. Indeed, I suspect that's all we ever do.

But even if I'm wrong, and it turns out that there really are terminal values in there somewhere, then I expect that most humans aren't aware of them and if some external system starts optimizing for them, and is willing to trade arbitrary amounts of a merely instrumental good in exchange for the terminal good it serves as a proxy for (as well it should), we'll experience that as emotionally unpleasant and challenging.

Comment author: Ultima 24 January 2011 08:23:32PM 1 point [-]

Solid answer, as far as I can see right now.