Eliezer showed a problem that that reasoning in his post on Pascal's Muggle.
Basically, human beings do not have an actual prior probability distribution. This should be obvious, since it means assigning a numerical probability to every possible state of affairs. No human being has ever done this, or ever will.
But you have something like a prior, but you build the prior itself based on your experience. At the moment we don't have a specific number for the probability of the mugging situation coming up, but just think it's very improbable, so that we don't expect any evidence to ever come up that would convince us. But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
You might say that means it was already that high, but it does not mean this, given the objective fact that people do not have real priors.
Maybe humans don't really have probability distributions. But that doesn't help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.
Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn't be able to function in a probabilistic world if we didn't. But it's not relevant.
...But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough
Summary: the problem with Pascal's Mugging arguments is that, intuitively, some probabilities are just too small to care about. There might be a principled reason for ignoring some probabilities, namely that they violate an implicit assumption behind expected utility theory. This suggests a possible approach for formally defining a "probability small enough to ignore", though there's still a bit of arbitrariness in it.