jimrandomh comments on Making Beliefs Pay Rent (in Anticipated Experiences) - Less Wrong

110 Post author: Eliezer_Yudkowsky 28 July 2007 10:59PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (245)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: jimrandomh 01 March 2011 01:21:30PM 2 points [-]

Thanks for taking the time to try puzzling this out, but I suspect it's just interestingly wrong. The magic seems to be happening in this paragraph:

Joe prefers Option 1. Therefore he anticipates that he will choose Option 1. Therefore, his current utility is 2U(1/2). But what if he anticipated that he would choose Option 2? Then his current utility would be 2U(1/2+p/2). So he wishes his k were smaller than U-inverse(k), meaning he wishes his U(x) were closer to xU(1). If he were to modify his utility function such that U'(x) = xU(1) for all x, the new Joe would not regret this decision since it strictly increases his expected utility under the new function.

I don't see where U(1/2+p/2) comes from; should that be U(1)+U(p)? I'm also not sure it's possible for the agent to anticipate choosing option 2, given the information it has. Finally, what does it matter whether a change increases expected utility under the new function? It's only utility under the old function that matters - changing utility function to almost anything maximizes the new function, including degenerate utility functions like number of paperclips.

Comment author: HonoreDB 02 March 2011 12:02:36AM *  0 points [-]

I don't see where U(1/2+p/2) comes from

Joe doesn't know yet which proposition would get 1 and which would get p, so he assigns the average to both. He anticipates learning which is which, at which point it would change to 1 and p.

I'm also not sure it's possible for the agent to anticipate choosing option 2, given the information it has.

Not sure what you mean here.

Finally, what does it matter whether a change increases expected utility under the new function?

It just shows the asymmetry. Joe can maximize U by changing into Joe-with-U', but Joe-with-U' can't maximize U' by changing back to U.