A green sky will be green. A pink invisible unicorn is pink. A moral system would be moral. All tautologies, none are true.

I still wonder, however, if there is any possible rational reason to not choose to sacrifice oneself in the scenario.

Of course there is -- e.g. if you care more for yourself than for other people, rationality doesn't compel you to sacrifice even a cent of your money, let alone you life, for the sake of others.

People must REALLY REALLY stop confusing what is "rational" and what is "moral". Rationality says nothing about what you value, only about how to achieve it.

They must also stop confusing "should" "would" and "I would prefer to".

Myself, I am not a utilitarian, but a deontologist. I would flip the switch, because I have been given the choice to choose between two different losses, inescapably, and I would try to minimize this loss. As for pushing someone else in front of the trolley, I could not abide someone doing that to me or a loved one, throwing us from relative safety into absolute disaster. So I would not do it to another. It is not my sacrifice to make.

As for throwing myself in front of the trolley...

I would want to. In the calm state I am in right now, I would do it. In the moment, there is a more than probable chance that fear will take hold and I would not sacrifice myself for five others. But in this scenario, I would probably be too stressed to think to throw another person into a train, let alone myself. So if we are taking the effects of stress out of my cognitive calculations, I will take the effects of stress out of my moral calculations.

I would do it.

only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were 'in the lottery', all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal's muggings occur

How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?

That's part of why I dislike Robin Hanson's original solution. That the tempting/blackmailing offer involves 3^^^^3 other people, and that you are also a person should be merely incidental to one particular illustration of the problem of Pascal's Mugging -- and as such it can't be part of a solution to the core problem.

To replace this with something like "causal nodes", as Eliezer mentions, might perhaps solve the problem. But I wish that we started talking about Clippy and his paperclips instead, so that the original illustration of the problem which involves incidental symmetries doesn't mislead us into a "solution" overreliant on symmetries.

Random thoughts here, not highly confident in their correctness.

Why is the leverage penalty seen as something that needs to be added, isn't it just the obviously correct way to do probability.

Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.0000007%.

If I then declare this to be my probability of become Overlord of Earth in an alien-backed coup, this is obviously wrong. Clearly I should multiply it by the probability that the aliens pick me, given that the aliens are doing this. There are about 7-billion people on earth, and updating on the existence of Overlord Declaring aliens doesn't have much effect on that estimate, so my probability of being picked is about 1 in 7 billion, meaning my probability of being overlorded is about 0.0000000000000001%. Taking the former estimate rather than the latter is simply wrong.

Pascal's mugging is a similar situation, only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were 'in the lottery', all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal's muggings occur (which should be very small but not super-exponentially small). This gives you the leverage penalty straight away, no need to think about Tegmark multiverses. We were simply mistaken to not include it in the first place.

I agree with your analysis regarding the difference between systematic naming systems and merely similar naming. That said, the justification for more clearly separating Pascal's mugging and this other unnamed situation does strike me as a political decision or rationalization. If the real world impact of people's misunderstanding were beneficial for the AI friendly cause, I doubt if anyone here would be making much ado about it. I would be in favor of renaming moissanite to diamand if this would help avert our ongoing malinvestment in clear glittery rocks to the tune of billions of dollars and numerous lives, so political reasons can perhaps be justified in some situations.

Hm, that's a good point, I've changed my opinion about this case.

When I wrote my comment, I was thinking primarily of words that share a common prefix or suffix, which tends to imply that they refer to things that share the same category but are not the same thing. "English" and "Spanish", for example.

But yeah, "diyer" is too close to "die" to be easily distinguishable. Maybe "rubemond"?

Chemistry would not be improved by providing completely different names to chlorate and perchlorate (e.g. chlorate and sneblobs). Also, I think English might be better if rubies were called diyermands. If all of the gemstones were named something that followed a scheme similar to diamonds, that might be an improvement.

Would the English language really be better if rubies were called diyermands?

I suspect it would be. The first time one encounters the word "ruby", you have only context to go off of. But if the word sounded like "diamond", then you could also make a tentative guess that the referent is also similar.

If someone suggests to me that they have the ability to save 3^^^3 lives, and I assign this a 1/3^^^3 probability, and then they open a gap in the sky at billions to one odds, I would conclude that it is still extremely unlikely that they can save 3^^^3 lives. However, it is possible that their original statement is false and yet it would be worth giving them five dollars because they would save a billion lives. Of course, this would require further assumptions on whether people are likely to do things that they have not said they would do, but are weaker versions of things they did say they would do but are not capable of.

Also, I would assign lower probabilities when they claim they could save more people, for reasons that have nothing to do with complexity. For instance, "the more powerful a being is, the less likely he would be interested in five dollars" or :"a fraudster would wish to specify a large number to increase the chance that his fraud succeeds when used on ordinary utility maximizers, so the larger the number, the greater the comparative likelihood that the person is fraudulent".

the phrase "Pascal's Mugging" has been completely bastardized to refer to an emotional feeling of being mugged that some people apparently get when a high-stakes charitable proposition is presented to them, regardless of whether it's supposed to have a low probability.

1) Sometimes what you may actually be seeing is disagreement on whether the hypothesis has a low probability.

2) Some of the arguments against Pascal's Wager and Pascal's Mugging don't depend on the probability. For instance, Pascal's Wager has the "worshipping the wrong god" problem--what if there's a god who prefers that he not be worshipped and damns worshippers to Hell? Even if there's a 99% chance of a god existing, this is still a legitimate objection (unless you want to say there's a 99% chance specifically of one type of god).

3) In some cases, it may be technically true that there is no low probability involved but there may be some other small number that the size of the benefit is multiplied by. For instance, most people discount events that happen far in the future. A highly beneficial event that happens far in the future would have the benefit multiplied by a very small number when considering discounting.

Of course in cases 2 and 3 that is not technically Pascal's mugging by the original definition, but I would suggest the definition should be extended to include such cases. Even if not, they should at least be called something that acknowledges the similarity, like "Pascal-like muggings".