= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Extreme risks: when not to use expected utility - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (36)
A thought on Pascal's Mugging:
One source of "the problem" seems to be a disguised version of unbounded payoffs.
Mugger: I can give you any finite amount of utility.
Victim: I find that highly unlikely.
Mugger: How unlikely?
Victim: 1/(really big number)
Mugger: Well, if you give me $1, I'll give you (really big number)^2 times the utility of one dollar. Then your expected utility is positive, so you should give me the money.
The problem here is that whatever probability you give, the Mugger can always just make a better promise. Trying to assign "I can give you any finite amount of utility" a fixed non-zero probability is equivalent to assigning "I can give you an infinite amount of utility" a fixed non-zero probability. It's sneaking an infinity in through the back door, so to speak.
It's also very hard for any decision theory to deal with the problem "Name any rational number, and you get that much utility." That's because there is no largest rational number; no matter what number you name, there is another number that it is better to name. We can even come up with a version that even someone with a bounded utility function can be stumped by; "Name any rational number less than ten, and you get that much utility." 9.9 is dominated by 9.99, which is dominated by 9.999, and so on. As long as you're being asked to choose from a set that doesn't contain its least upper bound, every choice is strictly dominated by some other choice. Even if all the numbers involved are finite, being given an infinite number of options can be enough to give decision theories the fits.
Interesting thoughts on the mugger. But you still need a theory able to deal with it, not just an understanding of the problems.
For the second part, you can get a good decision theory for the "Name any rational number less than ten, and you get that much utility," by giving you a certain fraction of negutility for each digit of your definition; there comes a time when the time wasted adding extra '9's dwarfs the gain in utility. See Tolstoy's story How Much Land Does a Man Need for a traditional literary take on this problem.
The "Name any rational number, and you get that much utility" problem is more tricky, and would be a version of the "it is rational to spend infinity in hell" problem. Basically if your action (staying in hell; or specifying your utility) give you more ultimate utility than you lose by doing so, you will spend eternity doing your utility-losing action, and never cash in on your gained utility.