According to orthodox expected utility theory, the boundedness of the utility function follows from standard decision-theoretic assumptions, like Savage's fairly weak axioms or the von Neumann-Morgenstern continuity/the Archimedean property axiom. Unbounded expected utility maximization violates the sure-thing principle, is vulnerable to Dutch books and is vulnerable to money pumps, all plausibly irrational. See, for example, Paul Christiano's comment with St. Petersburg lotteries (and my response). So, it's pretty plausible that unbounded expected utility maximization is just inevitably formally irrational.
However, I'm not totally sure, since there are some parallels to Newcomb's problem and Parfit's hitchhiker: you'd like to precommit to following a rule ahead of time that leads to the best prospects, but once some event happens, you'd like to break the rule and maximize local value greedily instead. But breaking the rule means you'll end up with worse prospects over the whole sequence of events than if you had followed it. The rules are:
- Newcomb's problem: taking the one box
- Parfit's hitchhiker: paying back the driver
- Christiano's St. Petersburg lotteries: sticking with the best St. Petersburg lottery offered
So, rather than necessarily undermining unbounded expected utility maximization, maybe this is just a problem for "local" expected utility maximization, since there are other reasons you want to be able to precommit to rules, even if you expect to want to be able to break them later. Having to make precommitments shouldn't be decisive against a decision theory.
Still, it seems better to avoid precommitments when possible because they're messy, risky and ad hoc. Bounded utility functions seem like a safer and cleaner solution here; we get a formal proof that they work in idealized scenarios. I also don't even know if precommitments generally solve unbounded utility functions' apparent violations of decision-theoretic principles that bounded utility functions don't have; I may be generalizing too much from one case.
I notice that Savage's axioms require you to have consistent preferences over an unreasonably broad set of actions, namely any state-outcome relationship that could mathematically exist, even if it is completely and extremely physically impossible.
I think that's an extremely strong decision-theoretic assumption.
One notion that deconfused these sorts of incredibly low probabilities to me is to just do a case split.
Suppose we have a cup of coffee. Probably if you drink it, nothing much happens. But by Cromwell it is conceivable that it was actually planted by an Eldritch trickster god and that if you drink it the Eldritch trickster god will torture 3^^^^^3 people for 100 years.
Now obviously the trickster god scenario is very unlikely, I'd say much less than 1e-1000000 probability. (IMO think we should have at least as many zeros as I used of characters to describe ... (read more)