This post describes an infinite gamble that, under some reasonable assumptions, will motivate people who act to maximize an unbounded utility function to send me all their money. In other words, if you understand this post and it doesn't motivate you to send me all your money, then you have a bounded utility function, or perhaps even upon reflection you are not choosing your actions to maximize expected utility, or perhaps you found a flaw in this post.
Briefly, we do this with The St. Petersburg Paradox, converted to a mugging along the lines of Pascal's Mugging. I then tweaked it to extract all of the money instead of just a fixed sum.
I have always wondered if any actual payments have resulted from Pascal's Mugging, so I intend to track payments received for this variation. If anyone does have unbounded utility and wants to prove me wrong by sending money, send it with Paypal to tim at fungible dot com. Annotate the transfer with the phrase "St. Petersburg Mugging", and I'll edit this article periodically to say how much money I received. In order to avoid confusing the experiment, and to exercise my spite, I promise I will not spend the money on anything you will find especially valuable. SIAI would be better charity, if you want to do charity, but don't send that money to me.
Here's the hypothetical (that is, false) offer to persons with unbounded utility:
- Let's call your utility function "UTILITY". We assume it takes a state of the universe as an argument.
- Define DUT to be UTILITY(the present situation plus you receiving $1000)-UTILITY(the present situation). Here DUT stands for Difference in UTility. We assume DUT is positive.
- You have unbounded utility, so for each nonnegative N there is a universe UN(N) such that UTILITY(UN(N)) is at least DUT * 2**N. Here UN stands for "universe".
- The phrase "I am a god" is defined to mean that I am able to change the universe to any state I choose. I may not be a god after I make the change.
- The offer is: For every dollar you send me, I will flip a coin. If it comes out Tails, or I am not a god, I will do nothing. If it comes out Heads and I am a god, I will flip the coin repeatedly until I see it come up Heads again. Let T be the number of times it was Tails. I will then change the universe to UN(T).
If I am lying and the offer is real, and I am a god, what utility will you receive from sending me a dollar? Well, the probability of me seeing N Tails followed by a Head is (1/2)**(N + 1), and your utility for the resulting universe is UTILITY(UN(N)) >= DUT * 2**N, so your expected utility if I see N tails is (1/2)**(N + 1) * UTILITY(UN(N)) >= (1/2)**(N + 1) * DUT * 2 ** N = DUT/2. There are infinitely many possible values for N, so your total expected utility is positive infinity * DUT/2, which is positive infinity.
I hope we agree that it is unlikely that I am a god, but it's consistent with what you have observed so far, so unless you were born with certain knowledge that I am not a god, you have to assign positive probability to it. Similarly, the probability that I'm lying and the above offer is real is also positive. The product of two positive numbers is positive. Combining this with the result from the previous paragraph, your expected utility from sending me a dollar is infinitely positive.
If you send me one dollar, there will probably be no result. Perhaps I am a god, and the above offer is real, but I didn't do anything beyond flipping the first coin because it came out Tails. In that case, nothing happens. Your expected utility for the next dollar is also infinitely positive, so you should send the next dollar too. By induction you should send me all your dollars.
If you don't send money because you have bounded utility, that's my desired outcome. If you do feel motivated to send me money, well, I suppose I lost the argument. Remember to send all of it, and remember that you can always send me more later.
As of 7 June 2011, nobody has sent me any money for this.
ETA: Some interesting issues keep coming up. I'll put them here to decrease the redundancy:
- Yes, you can justify not giving me money because I might be a god by claiming that there are lots of other unlikely gods that have a better claim on your resources. My purpose in writing this post is to find a good reason not to be jerked around by unlikely gods in general. Finding a reason to be jerked around by some other unlikely god is missing the point.
- I forgot to mention that if I am a god, I can stop time while I flip coins, so we aren't resource-constrained on the number of times I can flip the coin.
- Yes, you can say that your prior probability of me being a god is zero. If you want to go that way, can you say what that prior probability distribution looks like in general? I'm actually more worried about making a Friendly AI that gets jerked around by an unlikely god that we did not plan for, so having a special case about me being god doesn't solve an interesting portion of the problem. For what it's worth, I believe the Universal Prior would give positive small probability to many scenarios that have a god, since universes with a god are not incredibly much more complex than universes that don't have a god.
A bigger problem is your ability to hand out arbitrarily large amounts of utility. Suppose the universe can be simulated by an N state Turing machine, this limits the number of possible states it can occupy to a finite (but probably very large) number. This in turn bounds the amount of utility you can offer me, since each state has finite utility and the maximum of a finite set of finite numbers is finite. (The reason why this doesn't automatically imply a bounded utility function is that we are uncertain of N.)
As a result of this:
P(you can offer me k utility) > 0 for any fixed k
but
P(you can offer me x utility for any x) = 0
To be honest thought, I'm not really comfortable with this, and I think Solomonoff needs to be fixed (I don't feel like I believe with certainty that the universe is computable). The real reason why you haven't seen any of my money is that I think the maths is bullshit, as I have mentioned elsewhere.
Thinking about it more, this isn't a serious problem for the dilemma. While P(you can offer me k utility) goes to zero as k goes to infinity but there's no reason to suppose it goes faster then 1/n does.
This means you can still set a similar dilemma, with a probability of you being able to offer me 2^n utility eventually becoming greater than (1/2)^n for sufficiently large n, satisfying the conditions for a St Petersburg Lottery.