TheOtherDave comments on On the fragility of values - Less Wrong

4 Post author: Stuart_Armstrong 04 November 2011 06:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (31)

You are viewing a single comment's thread.

Comment author: TheOtherDave 04 November 2011 08:07:33PM 1 point [-]

I followed the first part of this, and I agree: if V=min(S1) and U=min(S2) and S2 = S1 + a, and a is not an instrumental value on the way to something else in S1, then V-maximization will cause a (and therefore U) to approach zero.

You lost me when you introduced W.

Your conclusion seems to be saying that a system that optimizes for certain things will reliably optimize for those things, the hard part is building a system that optimizes for the things we want. I agree with that much, certainly.

Comment author: Stuart_Armstrong 04 November 2011 08:52:31PM 0 points [-]

W is just a small error term on the utility function. A small error on S2 has a lot of consequences, an error on U has little.