The problem with Pascal's mugging is that it IS a fully general counterargument under classical decision theory. That's why it's a paradox right now. But saying "There's a problem with this paradox - therefore, I'll just ignore the problem' is not a solution.

I will argue that the prior for such a world should be of the order of 1/n or lower

This class of argument has been made before. The standard counterargument is that whatever argument you have for this conclusion, you cannot be 100% certain of its correctness. You should assign some nonzero probability to the hypothesis that the probability does not decrease fast enough for the correct expected utilities to be bounded. Then, taking this uncertainty into account, your expected utilities are unbounded.

The arguments relating to the bandwidth of our sensory system fail to account for (inefficient) encodings of that information which may have some configurations with arbitrarily low likelihood.

There is a positive lower bound to the probability of observing any given data (given a bound on the description length of the data), because you might just be getting random input. Given any observation that could be the result of some 1/3^^^^3 event, it could also just randomly pop into your brain for no reason with probability far greater than that. If you see a mechanism to output a random integer from 1 to 3^^^^3, and that its output was 7, you should be almost 100% confident that there was an error in your senses or your memory or your reasoning that convinced you that the mechanism works as described, etc (where "etc" means "anything other than that you observed the output of a mechanism that generates a random integer from 1 to 3^^^^3, and it was 7").

The point is that in this situation, just paying the mugger and carrying on cannot be the best course of action, because it's not the right choice if they're lying, and if they're not then it's dominated by other much larger considerations. Thus the mugging still fails, not necessarily because of the implausibility of their threat but because of the utter irrelevance of it in the face of unboundedly more important other considerations.

This totally fails to resolve the paradox. The conclusion that you should drop everything else and go all in on pursuing arbitrarily small probabilities of even more vast outcomes is, if anything, even more counter-intuitive than the conclusion that you should give the mugger $5.

Of course this doesn't really resolve the mugging itself. You could modify the scenario to replace myself having to pay with instead a small, plausible but entirely moral threat (e.g. "I'll punch that guy in the face"). I would then be motivated to make the correct moral decision regardless of bounds on my utility (though I suppose my motivation to be correct is itself bounded).

There is no reason that the "moral component" of your utility function must be linear. In fact, the boundedness of your utility function is the correct solution to Pascal's mugging.

If you view morality as entirely a means of civilisation co-ordinating then you're already immune to Pascal's Mugging because you don't have any reason to care in the slightest about simulated people who exist entirely outside the scope of your morality. So why bother talking about how to bound the utility of something to which you essentially assign zero utility to in the first place?

Or, to be a little more polite and turn the criticism around, if you do actually care a little bit about a large number of hypothetical extra-universal simulated beings, you need to find a different starting point for describing those feelings than facilitating civilisational co-ordination. In particular, the question of what sorts of probability trade-offs the existing population of earth would make (which seems to be the fundamental point of your argument) is informative, but far from the be-all and end-all of how to consider this topic.

This assumes you have no means of estimating how good your current secretary/partner is, other than directly comparing to past options. While it's nice to know what the optimal strategy is in that situation, don't forget that it's not an assumption which holds in practice.

The whole point of the Pascal's Mugging scenario is that the probability doesn't decrease faster than the reward. If for example, you decrease the probability by half for each additional bit it takes to describe, 3^^^3 still only takes a few bits to write down.

Do you believe it's literally impossible that there is a matrix? Or that it can't be 3^^^3 large? Because when you assign these things so low probability, you are basically saying they are impossible. No amount of evidence could convince you otherwise.

I think EY had the best counter argument. He had a fictional scenario where a physicist proposed a new theory that was simple and fit the data perfectly. But the theory also implies a new law of physics that could be exploited for computing power, and would allow unfathomably large amounts of computing power. And that computing power could be used to create simulated humans.

Therefore anyone alive today has a small probability of affecting large amounts of simulated people. Since that is impossible, the theory must be wrong. It doesn't matter if it's simple or if it fits the data perfectly.

If they're finite then the game has a finite value, you can calculate it, and there's no paradox. In which case median utility can only give the same answer or an exploitably wrong answer.

Even in finite case, I believe it can grow quite large as the number of iterations increases. It's one expected dollar each step. Each step having half the probability of the previous step, and twice the reward.

Imagine the game goes for n finite steps. An expected utility maximizer would still spend $n to play the game. A median maximizer would say "You are never going to win in the liftetime of the universe and then some, so no thanks." The median maximizer seems correct to me.

Median utility does fail trivially. But it opens the door to other systems which might not. He just posted a refinement on this idea, Mean of Quantiles.

IMO this system is much more robust than expected utility. EU is required to trade away utility from the majority of possible outcomes to really rare outliers, like the mugger. Median utility will get you better outcomes at least 50% of the time. And tradeoffs like the one above, will get you outcomes that are good in the majority of possible outcomes, ignoring rare outliers. I'm not satisfied it's the best possible system, so the subject is still worth thinking about and debating.

I don't think any of your paradoxes are solved. You can't get around Pascal's mugging by modifying your probability distribution. The probability distribution has nothing to do with your utility function or decision theory. Besides being totally inelegant and hacky, there might be practical consequences. Like you can't believe in the singularity now. The singularity could lead to vastly high utility futures, or really negative ones. Therefore it's probability must be extremely small.

The St Petersburg casino is silly of course, but there's no reason a real thing couldn't produce a similar distribution. If you have some sequence of probabilities dependent on each other, that each have 1/2 probability, and give increasing utility.

This seems to be a case of trying to find easy solutions to hard abstract problems at the cost of failing to be correct on easy and ordinary ones. It's also fairly trivial to come up with abstract scenarios where this fails catastrophically, so it's not like this wins on the abstract scenarios front either. It just fails on a new and different set of problems - ones that aren't talked about because no-one's ever found a way to fail on them before.

Also, all of the problems you list it solving are problems which I would consider to be satisfactorily solved already. Pascal's mugging fails if the believability of the claim is impacted by the magnitude of the numbers in it, since the mugger can keep naming bigger numbers and simply suffer lower credibility as a result. The St Petersburg paradox is intellectually interesting but impossible to actually construct in practice given a finite universe (versions using infinite time are defeated by bounded utility within a time period and geometric future discounting). The Cauchy distribution is just one of many functions with no mean, all that tells me is that it's the wrong function to model the world with if you know the world should have a mean. And the repungent conclusion, well I can't comment usefully about this because "repungent" or not I've never viewed it to be incorrect in the first place - so to me this potentially justifying smaller but happier populations is an error if anything.

I just think it's worth making the point that the existing, complex solutions to these problems are a good thing. Complexity-influenced priors, careful handling of infinite numbers, bounded utility within a time period, geometric future discounting, integratable functions and correct utility summation and zero-points are all things we want to be doing anyway. Even when they're not resolving a paradox! The paradoxes are good, they teach us things which circumventing the paradoxes in this way would not.

PS People feel free to correct my incomplete resolutions of those paradoxes, but be mindful of whether any errors or differences of opinion I might have actually undermine my point here or not.

When I click main it defaults to "promoted", you linked in this comment to main/all which caught the post. I wonder if anyone else has the problem of missing posts like this. There is no way I could have found this post if I didn't try really hard to work out where it was. After 10 mins I decided to screenshot and ask someone..

Is there something that can be done about these posts being lost/not-easy to find?

My usual process is:

1. www.lesswrong.com
2. -discusssion
3. (sometimes click >main)
4. sometimes autocomplete to http://lesswrong.com/r/discussion/new/

I can easily change what I do to also check Main/notpromotedbutnormalmainposts. But I wonder if anyone else misses things and can be helped like this?

Meta comment (I can PM my actual responses when I work out what I want them to be); I found I really struggled with this process, because of the awkward tension between answering the questions and playing a role. I just don't understand what my goal is.

Let me call my view position 1, and the other view position A. The first time I read just this post and I thought it was just a survey, where I should "give my honest opinion", but where some of the position A questions would be non-sensical for someone of position 1 so just pretend a little in order to give an answer that's not "mu".

Then I read the link on what an Ideological Turing test actually was, and that changed my thinking completely. I don't want to give almost-honest answers to position A. I want to create a character who is a genuinely in position A and write entirely fake answers that are as believable as possible and may have nothing to do with my opinions.

In my first attempt at that though, it was still obvious which was which, because my actual views for position 1 were nuanced, unusual and contained a fair number of pro-A elements, making it quite clear when I was giving my actual opinion. So I start meta-gaming. If I want to fool people I really want a fake position 1 opinion as well. In fact if I really want to fool people I need to create a complete character with views nothing like my own, and answer as them for both sets. But surely anyone could get 50% by just writing obviously ignorant answers for both sides? Which doesn't seem productive.

I guess my question is, what's my "win" condition here? Are we taking individuals and trying to classify their position? If so do I "win" if it's 50-50, or do I "win" if it's 100-0 in favour of the opposite opinion? Or are we mixing all the answers for position A and then classifying them as genuine or fake, then separately doing the same for position 1? In that case I suppose I "win" if the position I support is the one classified with higher accuracy. In other words I want to get classified as genuine twice. That actually makes the most sense, maybe I'm just getting confused by all the paired-by-individual responses in the comments, which is not at all how the evaluators will see it, they should not be told which pairs are from the same person at all.

Sorry maybe everyone else gets this already, but I would have thought there's others reading just this post without enough context who might have similar issues.

I think this shows how the whole "language independent up to a constant" thing is basically just a massive cop-out. It's very clever for demonstrating that complexity is a real, definable thing, with properties which at least transcend representation in the infinite limit. But as you show it's useless for doing anything practical.

My personal view is that there's a true universal measure of complexity which AIXI ought to be using, and which wouldn't have these problems. It may well be unknowable, but AIXI is intractable anyway so what's the difference? In my opinion, this complexity measure could give a real, numeric answer to seemingly stupid questions like "You see a number. How likely is it that the number is 1 (given no other information)?". Or it could tell us that 16 is actually less complex than, say, 13. I mean really, it's 2^2^2, spurning even a need for brackets. I'm almost certain it would show up in real life more often than 13, and yet who can even show me a non-contrived language or machine in which it's simpler?

Incidentally, they "hell" scenario you describe isn't as unlikely as it at first sounds. I remember an article here a while back lamenting the fact that left unmonitored AIXI could easily kill itself with exploration, the result of which would have a very similar reward profile to what you describe as "hell". It seems like it's both too cautious and not cautious enough in even just this one scenario.