Given the mixed strategy, taking and not taking your bet both have infinite expected utility, even if there are no other infinite expected utility lotteries.
I agree that there are many bets with infinite expected utility for a person who has unbounded utility. If the subject takes those bets into account, it's unlikely that I'll win in the sense of getting the subject to send me money. However, if the subject takes them into account, it's very likely that the subject will lose, in the sense that the subject's estimated utility from all of these infinite expected utility bets is going to swamp utility from ordinary things. If someone is hungry and has an apple, they should pay attention to the apple to decide whether to eat it, and not pay attention to the relative risk of Tim being a god who disapproves of apple-eating or Carl being a god who promotes apple-eating.
2) To get a decision theory that actually would take infinite expected utility lotteries with high probability we would need to use something like the hyperreals, which would allow for differences in the utility of different probabilities of infinite payoff. But once we do that, the fact that your offer is so implausible penalizes it.
I don't care much whether someone accepts my offer. I really care whether they can pay attention to an apple when they're hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple. I am not convinced that hyperreals solve that problem -- so far as I can tell, the outcome would be unchanged. Can you explain why you think hyperreals might help?
(ETA: hyperreals, AKA the non-standard reals, aren't mysterious. Imagine the real numbers, imagine a new one we might call "infinity", then add other new numbers as required so all of the usual first-order properties still hold. So we'd have infinity - 3 and 5 * infinity - 376/infinity and so forth. So far as I can tell, if you do the procedure described in the OP with hyperreal utilities, you still conclude that the utility of giving me money exceeds the utility of keeping the money and spending it on something ordinary.)
Conditional on there being any sources of infinite utility, it is far more likely that they will be better obtained by other routes than by succumbing to this trick.
Perhaps you meant unbounded instead of infinite there.
I'm concerned that the tricky routes will dominate the non-tricky routes. I don't really expect anyone to fall for my specific trick.
Also see Nick Bostrom's infinitarian ethics paper.
Last I read that was long ago. I glanced at it just now and it seems to be concerned with ethics, rather than an individual deciding what to do, so I'm having doubts about it being directly relevant. It's probably worth looking at anyway, but if you can say specifically how it's relevant and cite a specific page it would help.
The problem here is technical, in the construction of your example.
Does the problem still exist if we assume the purpose of my example is to show that people with unbounded utility lose, rather than to make people send me money?
I really care whether they can pay attention to an apple when they're hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple.
Personal survival makes it a lot easier to please unlikely gods, so eating the apple is preferred. For more general situations, some paths to infinity are much more probable than others. For example, perhaps we can build a god.
This post describes an infinite gamble that, under some reasonable assumptions, will motivate people who act to maximize an unbounded utility function to send me all their money. In other words, if you understand this post and it doesn't motivate you to send me all your money, then you have a bounded utility function, or perhaps even upon reflection you are not choosing your actions to maximize expected utility, or perhaps you found a flaw in this post.
Briefly, we do this with The St. Petersburg Paradox, converted to a mugging along the lines of Pascal's Mugging. I then tweaked it to extract all of the money instead of just a fixed sum.
I have always wondered if any actual payments have resulted from Pascal's Mugging, so I intend to track payments received for this variation. If anyone does have unbounded utility and wants to prove me wrong by sending money, send it with Paypal to tim at fungible dot com. Annotate the transfer with the phrase "St. Petersburg Mugging", and I'll edit this article periodically to say how much money I received. In order to avoid confusing the experiment, and to exercise my spite, I promise I will not spend the money on anything you will find especially valuable. SIAI would be better charity, if you want to do charity, but don't send that money to me.
Here's the hypothetical (that is, false) offer to persons with unbounded utility:
If I am lying and the offer is real, and I am a god, what utility will you receive from sending me a dollar? Well, the probability of me seeing N Tails followed by a Head is (1/2)**(N + 1), and your utility for the resulting universe is UTILITY(UN(N)) >= DUT * 2**N, so your expected utility if I see N tails is (1/2)**(N + 1) * UTILITY(UN(N)) >= (1/2)**(N + 1) * DUT * 2 ** N = DUT/2. There are infinitely many possible values for N, so your total expected utility is positive infinity * DUT/2, which is positive infinity.
I hope we agree that it is unlikely that I am a god, but it's consistent with what you have observed so far, so unless you were born with certain knowledge that I am not a god, you have to assign positive probability to it. Similarly, the probability that I'm lying and the above offer is real is also positive. The product of two positive numbers is positive. Combining this with the result from the previous paragraph, your expected utility from sending me a dollar is infinitely positive.
If you send me one dollar, there will probably be no result. Perhaps I am a god, and the above offer is real, but I didn't do anything beyond flipping the first coin because it came out Tails. In that case, nothing happens. Your expected utility for the next dollar is also infinitely positive, so you should send the next dollar too. By induction you should send me all your dollars.
If you don't send money because you have bounded utility, that's my desired outcome. If you do feel motivated to send me money, well, I suppose I lost the argument. Remember to send all of it, and remember that you can always send me more later.
As of 7 June 2011, nobody has sent me any money for this.
ETA: Some interesting issues keep coming up. I'll put them here to decrease the redundancy: