I stumbled across this fix and unfortunately discovered what I consider to be a massive problem with it - it would imply that your utility function is non-computable.
OK. So in order for this to work, it needs to be the case that your prior has the property that: P(3^^^3 disutility | I fail to give him $5) << 1/3^^^3.
Unfortunately, if we have an honest Kolmogorov prior and utility is computable via a complexity << 3^^^3 Turing machine, this cannot possibly be the case. In particular, it is a Theorem that for any computable function C (whose Turing machine has complexity K(C)), so that there are x with C(x) > N, then under the Kolmogorov prior for x we have that: P( C(x) > N ) >> 2^{ - K(C) - K(N) } Now, since K(3^^^3) is small, as long as utility is computed by a small Turing machine, and it is possible to have 3^^^3 disutility, such a circumstance will not be too unlikely under a Kolmogorov prior.
For those interested, here's how the theorem is proved. I will produce a Turing machine of size K(C) + K(N) +O(1) that outputs an x (in fact, the smallest x) so that C(x) > N. By definition, I can encode C and N in size K(C) + K(N) +O(1). I then have a Turing machine enumerate all x until it finds one so that C(x) > N and output's that x. This provides a lower bound.
I guess the problem is that if you just have a Kolmogorov prior, there is a relatively simple universe that is actually out to get you. In fact, being the shortest computation causing 3^^^3 disutility is actually a pretty simple condition.
For background, see here.
In a comment on the original Pascal's mugging post, Nick Tarleton writes:
Coming across this again recently, it occurred to me that there might be a way to generalize Vassar's suggestion in such a way as to deal with Tarleton's more abstract formulation of the problem. I'm curious about the extent to which folks have thought about this. (Looking further through the comments on the original post, I found essentially the same idea in a comment by g, but it wasn't discussed further.)
The idea is that the Kolmogorov complexity of "3^^^^3 units of disutility" should be much higher than the Kolmogorov complexity of the number 3^^^^3. That is, the utility function should grow only according to the complexity of the scenario being evaluated, and not (say) linearly in the number of people involved. Furthermore, the domain of the utility function should consist of low-level descriptions of the state of the world, which won't refer directly to words uttered by muggers, in such a way that a mere discussion of "3^^^^3 units of disutility" by a mugger will not typically be (anywhere near) enough evidence to promote an actual "3^^^^3-disutilon" hypothesis to attention.
This seems to imply that the intuition responsible for the problem is a kind of fake simplicity, ignoring the complexity of value (negative value in this case). A confusion of levels also appears implicated (talking about utility does not itself significantly affect utility; you don't suddenly make 3^^^^3-disutilon scenarios probable by talking about "3^^^^3 disutilons").
What do folks think of this? Any obvious problems?