Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

gwern comments on Extreme risks: when not to use expected utility - Less Wrong

4 Post author: Stuart_Armstrong 23 October 2009 02:40PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (36)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 24 October 2009 09:49:28PM 1 point [-]

It's sneaking an infinity in through the back door, so to speak.

Yes, this is precisely my own thinking - in order to give any assessment of the probability of the mugger delivering on any deal, you are in effect giving an assessment on an infinite number of deals (from 0 to infinity), and if you assign a non-zero probability to all of them (no matter how low), then you wind up with nonsensical results.

Giving the probability beforehand looks even worse if you ignore the deal aspect and simply ask what is the probability that anything the mugger says would be true? (Since this includes as a subset any promises to deliver utils.) Since he could make statements about turing machines or Chaitin's Omega etc., now you're into areas of intractable or undecidable questions!

As it happens, 2 or 3 days ago I emailed Bostrom about this. There was a followup paper to Bostrom's "Pascal's Mugging", also published in Analysis, by a Baumann, who likewise rejected the prior probability, but Baumann didn't have a good argument against it but to say that any such probability is 'implausible'. Showing how infinities and undecidability get smuggled into the mugging shores up Baumann's dismissal.

But once we've dismissed the prior probability, we still need to do something once the mugger has made a specific offer. If our probability doesn't shrink at least as quickly as his offer increases, then we can still be mugged; if it shrinks exactly as quickly or even more quickly, we need to justify our specific shrinkage rate. And that is the perplexity: how fast do we shrink, and why?

(We want the Right theory & justification, not just one that is modeled after fallible humans or ad hocly makes the mugger go away. That is what I am asking for in the toplevel comment.)