Eliezer,

OK, one more try. First, you're picking 3^^^^3 out of the air, so I don't see why you can't pick 1/3^^^^3 out of the air also. You're saying that your priors have to come from some rigorous procedure but your utility comes from simply transcribing what some dude says to you. Second, even if for some reason you really want to work with the utility of 3^^^^3, there's no good reason for you not to consider the possibility that it's really -3^^^^3, and so you should be doing the opposite. The issue is not that two huge numbers will exactly cancel out; the point is that you're making up _all_ the numbers here but are artificially constraining the expected utility differential to be positive.

If I really wanted to consider this example realistically, I'd say that this guy has no magic powers, so I wouldn't worry about him killing 3^^^^3 people or whatever. A slightly more realistic scenario would be something like a guy with a bomb in a school, in which case I'd defer to the experts (presumably whoever in the police force deals with people like that) on their judgment of how best to calm him down. There I could see an (approximate) probability calculation being relevant, but, again, they key thing would be whether giving him $5 (or whatever) would make him more or less likely to set the fuse. It wouldn't be appropriate to say a priori that it could only help.