TheMajor comments on Proper value learning through indifference - Less Wrong

16 Post author: Stuart_Armstrong 19 June 2014 09:39AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (50)

You are viewing a single comment's thread.

Comment author: TheMajor 19 June 2014 02:02:05PM 4 points [-]

Nice post! I hadn't read the previous posts on Cake or Death-like problems, but your post persuaded me to read them.

Enforcing indifference between choices to prevent abuse is a beautiful idea. You remark that as a consequence of this indifference the expected change in utility for changing from utility function 'v' to function 'w' is equal to 0, but isn't the converse also true (i.e. the desired update formula follows from the conservation of expected utility)? Above you justify/motivate the change in the update formula (marked in bold in your post) to E(w|v->w) on grounds of symmetry, but doesn't it simply follow from demanding that changing the utility function itself should not change the (average) utiliy? This sounds like a direct formalisation of demanding that our agent is indifferent with respect to its own utility function.

Comment author: Stuart_Armstrong 19 June 2014 02:15:19PM 1 point [-]

I think this is correct. When I write the principle up properly, I'll probably be emphasising that issue more.