PhilGoetz comments on Averaging value systems is worse than choosing one - Less Wrong

5 Post author: PhilGoetz 29 April 2010 02:51AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (56)

You are viewing a single comment's thread. Show more comments above.

Comment author: PhilGoetz 30 April 2010 12:44:20AM *  1 point [-]

Point taken - you certainly don't want to routinely solve problems by changing your values instead of changing your environment.

However, I think you tend to think about deep values, what I sometimes call latent values, while I often talk about surface values, of the type that show up in English sentences and in logical representations of them. People do change their surface values: they become vegetarian, quit smoking, go on a diet, realize they don't enjoy Pokemon anymore, and so on. I think that this surface-value-changing is well-modelled by energy minimization.

Whether there is a set of "deepest values" that never change is an open question. These are the things EY is talking about when he says an agent would never want to change its goals, and that you're talking about when you say an agent doesn't change its utility function. The EY-FAI model assumes such a thing exists, or that they should exist, or could exist. This needs to be thought about more. I think my comments in "Only humans can have human values" on "network concepts" are relevant. It's not obvious that a human's goal structure has top-level goals. It would be a possibly-unique exception among complex network systems if they do.