Maybe humans don't really have probability distributions. But that doesn't help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.

Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn't be able to function in a probabilistic world if we didn't. But it's not relevant.

But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.

That's just a regular bayesian probability update! You don't need to change terminology and call it something different.

At the moment we don't have a specific number for the probability of the mugging situation coming up, but just think it's very improbable, so that we don't expect any evidence to ever come up that would convince us.

That's fine. I too think the situation is extraordinarily implausible. Even Solomonoff induction would agree with us. The probability that the mugger is real would be something like 1/10^100. Or perhaps the exponent should be orders of magnitude larger than that. That's small enough that it shouldn't even remotely register as a plausible hypothesis in your mind. But big enough some amount of evidence could convince you.

You don't need to posit new models of how probability theory should work. Regular probability works fine at assigning really implausible hypotheses really low probability.

But that is still way, way bigger than 1/3^^^3.