I'm upvoting this post because it caused me to rethink some of my thoughts about utility and Pascal's mugging, something that very few of these posts [on Pascal's mugging] so far have done.
However, I am now somewhat confused. Doesn't the Archimedean property for utility (axiom P3' here) automatically imply that utility should be bounded? My argument for this would be as follows. I'm using the terminology from the Wikipedia page, but in case you don't want to click the link, u is utility, Eu is expected utility.
If Eu(N) was ever infinite, then there would be no epsilon > 0 such that (1-epsilon)L+epsilon N < M (here we start with some fixed L,M with L<M).
von Neumann utility is a statement about all lotteries, not just the ones induced by our beliefs. So in particular we have to consider the lottery A(E) where a fixed event E happens with probability 1. Since Eu(A(E)) is finite, and u(E) = Eu(A(E)), u(E) must be finite.
Finally, suppose that there was a sequence of events E1,E2,... whose utilities grew without bound. We can always pick out a subsequence F1,F2,... such that u(F(n+1)) >= 2u(F(n)) for all n. Then consider the lottery where Fn occurs with probability 2^(-n). This has infinite expected utility since its expected utility is sum(n=1 to infty) 2^(-n)u(F(n)) > sum(n=1 to infty) u(F(1)) = infty
Is there a flaw in this argument? Are the VNM utility axioms too strong (should preferences only need to be defined over lotteries that can actually be induced by updating our priors based on some amount of evidence; or alternately, is the Archimedean property too strong)?
Or is utility bounded? While I personally think that human preferences should lead to bounded utility functions, the conclusion that all rational agents must have bounded utility functions seems possibly too strong.
The vNM argument only uses preference over lotteries with finitely many possible outcomes to assign utilities to pure outcomes. One can use measure theory arguments to pin down the expected utilities of other lotteries. That some lotteries have infinite value does not seem to me to be a big problem.
Expected utility can be expressed as the sum ΣP(Xn)U(Xn). Suppose P(Xn) = 2-n, and U(Xn) = (-2)n/n. Then expected utility = Σ2-n(-2)n/n = Σ(-1)n/n = -1+1/2-1/3+1/4-... = -ln(2). Except there's no obvious order to add it. You could just as well say it's -1+1/2+1/4+1/6+1/8-1/3+1/10+1/12+1/14+1/16-1/5+... = 0. The sum depends on the order you add it. This is known as conditional convergence.
This is clearly something we want to avoid. Suppose my priors have an unconditionally convergent expected utility. This would mean that ΣP(Xn)|U(Xn)| converges. Now suppose I observe evidence Y. ΣP(Xn|Y)|U(Xn)| = Σ|U(Xn)|P(Xn∩Y)/P(Y) ≤ Σ|U(Xn)|P(Xn)/P(Y) = 1/P(Y)·ΣP(Xn)|U(Xn)|. As long as P(Y) is nonzero, this must also converge.
If my prior expected utility is unconditionally convergent, then given any finite amount of evidence, so is my posterior.
This means I only have to come up with a nice prior, and I'll never have to worry about evidence braking expected utility.
I suspect that this can be made even more powerful, and given any amount of evidence, finite or otherwise, I will almost surely have an unconditionally convergent posterior. Anyone want to prove it?
Now let's look at Pascal's Mugging. The problem here seems to be that someone could very easily give you an arbitrarily powerful threat. However, in order for expected utility to converge unconditionally, either carrying out the threat must get unlikely faster than the disutility increases, or the probability of the threat itself must get unlikely that fast. In other words, either someone threatening 3^^^3 people is so unlikely to carry it out to make it non-threatening, or the threat itself must be so difficult to make that you don't have to worry about it.