Wei_Dai comments on St. Petersburg Mugging Implies You Have Bounded Utility - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (163)
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
We are human and cannot really do that. With unbounded utilities, there are an absurdly large number of possible ways that an ordinary action can have very low-probability influence on a wide variety of very high-utility things, and you have to take them all into account and balance them properly to do the right thing. If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it's not likely to search all of the space. Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won't be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute. (As I said before, an improbable minor change to a likely god is an unlikely god, for the purposes of this discussion.)
If utilities are bounded, then the number of alternatives you have to look at doesn't grow so pathologically large, and you look at the apple to decide whether to eat the apple. The unlikely gods don't enter into it because you don't imagine that they can make enough of a difference to outweigh their unlikeliness.
Why can't they either estimate or prove that eating an apple has more expected utility (by please more gods overall than not eating an apple, say), without iterating over each god and considering them separately? And if for some reason you build an AI that does compute expected utility by brute force iteration of possibilities, then you obviously would not want it to consider only possibilities that "have been brought to their attention in the last minute". That's going to lead to trouble no matter what kind of utility function you give it.
(ETA: I think it's likely that humans do have bounded utility functions (if we can be said to have utility functions at all) but your arguments here are not very good. BTW, have you seen The Lifespan Dilemma?)