I keep seeing people responding to Pascal's Wager / Mugging by saying we just shouldn't pay attention to very low probabilities. (IIRC Eliezer said something similar as well). But intuitively I don't think this is true.
Imagine that a stranger comes up to you and offers you a cup of water and tells you, "Please drink this to humor me. It has a chemical I invented which seems to kill a tiny fraction of people in fascinatingly gruesome ways. But don't worry, I've already tested 10,000 people and only two of them have been affected." I have a hard time imagining that anybody would drink it.
The way I usually look at Pascal's Wager / Mugging is that this is just one of those paradoxes we seem to get into when we take decision theory and probability theory to extremes. There may or may not be a good decision-theoretic response to what precisely is the answer to the paradox, but in the meantime we need to go with our intuition that it's just wrong, even if we don't know exactly why.
So ultimately I think we just need to go with our intuitions on this. Is 1:10,000 too small a probability? What does your heart tell you?
Side note: I noticed that Nick Bostrom seems to invoke these sort of arguments several times in his book. I think he mentioned it regarding how much we should worry about x-risks especially given our astronomically large potential cosmic endowment. I also think he alluded to it when he mentions that a superintelligence would first take steps to prevent all sorts of catastrophic risks before proceeding with some new technologies.
I don't think that Pascal's mugging it's just a scenario where the expected payoff contains a small probability multiplied by an utility large in absolute value.
I think it is better to use the term "Pascal's mugging", or perhaps "Pascal scam" to describe a scenario where, in addition to the probabilities and the utilities being extreme, there is also lots of "Knightian" (= difficult to formalize) uncertainty about them.
Some people[1] are now using the term Pascal's mugging as a label for any scenario with a large associated payoff and a small or unstable probability estimate, a combination that can trigger the absurdity heuristic.
Consider the scenarios listed below: (a) Do these scenarios have something in common? (b) Are any of these scenarios cases of Pascal's mugging?
(1) Fundamental physical operations -- atomic movements, electron orbits, photon collisions, etc. -- could collectively deserve significant moral weight. The total number of atoms or particles is huge: even assigning a tiny fraction of human moral consideration to them or a tiny probability of them mattering morally will create a large expected moral value. [Source]
(2) Cooling something to a temperature close to absolute zero might be an existential risk. Given our ignorance we cannot rationally give zero probability to this possibility, and probably not even give it less than 1% (since that is about the natural lowest error rate of humans on anything). Anybody saying it is less likely than one in a million is likely very overconfident. [Source]
(3) GMOS might introduce “systemic risk” to the environment. The chance of ecocide, or the destruction of the environment and potentially humans, increases incrementally with each additional transgenic trait introduced into the environment. The downside risks are so hard to predict -- and so potentially bad -- that it is better to be safe than sorry. The benefits, no matter how great, do not merit even a tiny chance of an irreversible, catastrophic outcome. [Source]
(4) Each time you say abracadabra, 3^^^^3 simulations of humanity experience a positive singularity.
If you read up on any of the first three scenarios, by clicking on the provided links, you will notice that there are a bunch of arguments in support of these conjectures. And yet I feel that all three have something important in common with scenario four, which I would call a clear case of Pascal's mugging.
I offer three possibilities of what these and similar scenarios have in common:
In any case, I admit that it is possible that I just wanted to bring the first three scenarios to your attention. I stumbled upon each very recently and found them to be highly..."amusing".
[1] I am also guilty of doing this. But what exactly is wrong with using the term in that way? What's the highest probability for which the term is still applicable? Can you offer a better term?
[2] One would have to define what exactly counts as "direct empirical evidence". But I think that it is pretty intuitive that there exists a meaningful difference between the risk of an asteroid that has been spotted with telescopes and a risk that is solely supported by a priori arguments.