Updated case 2: Ask the person tied to the track what they would do if our situations were reversed. Do that. My reasoning is that the people on the track are much more affected by the decision than the people off of the track, and therefore my utility is probably maximized by letting them maximize their own utility. If I can hear the other 5 people, I'll let them take a vote and respect it.

Updated cases 3 and 4 are identical, it's just that humans think about actions kind of strangely. In both cases I would save myself unless I was feeling particularly depressed or altruistic at the moment.

The legality of updated case 4 may have been tested in motor vehicle accidents. Steering onto the sidewalk to avoid an oncoming runaway vehicle is virtually identical. My guess is that if there was no way to avoid either accident it would be difficult to show guilt for either choice.

I seem to recall some stories about train engineers who had to make the choice of staying on their runaway train to blow the warning whistle or try to react in other limited ways to save other people before an imminent collision instead of jumping to safety. From my fuzzy memory it seemed like engineers who jumped could be charged with something like dereliction of duty. Captains of ships have similar requirements for remaining with their vessel in distress even if it carries personal risk. So the answer to all of these questions may depend on whether the person who can flip the switch is working for the trolley company at the time.

The "jump in front of the trolley" question has the same problem. If I can stop the trolley by jumping in front of it, won't the first person tied to the tracks stop it as well?

Why can't we just state these hypothetical problems more directly?

Is it better that five people should die or that one person should die if there are no other options? I prefer the latter.

Is it better that five people should die or that I should directly kill one person if there are no other options? I prefer the latter.

Is it better that five people should die or that I should die if there are no other options? I prefer the former.

Is it better to value my own life more than arbitrarily many other people's lives? I don't think so.

His first, and seemingly most compelling, argument for Duplication over Unification is that, assuming an infinite universe, it's certain (with probability 1) that there is already an identical portion of the universe where you're torturing the person in front of you. Given Unification, it's meaningless to distinguish between that portion and this portion, given their physical identicalness, so torturing the person is morally blameless, as you're not increasing the number of unique observers being tortured.

I'd argue that the torture portion is not identical to the not-torture portion and that the difference is caused by at least one event in the common prior history of both portions of the universe where they diverged. Unification only makes counterfactual worlds real; it does not cause every agent to experience every counterfactual world. Agents are differentiated by the choices they make and agents who perform torture are not the same agents as those who abstain from torture. The difference can be made arbitrarily small, for instance by choosing an agent with a 50% probability of committing torture based on the outcome of a quantum coin flip, but the moral question in that case is why an agent would choose to become 50% likely to commit torture in the first place. Some counterfactual agents will choose to become 50% likely to commit torture, but they will be very different than the agents who are 1% likely to commit torture.

It may also be useful to identify the best thing. The difference between the best and worst is probably a useful measure of quality control as well as ensuring the tests are general enough to detect good as well as bad.

The spaceship has really good life support/recycling The spaceship is self-repairing and draws power from interstellar hydrogen

That requires a MTTF of 3^^^3 years, or a per-year probability of failure of roughly 1/3^^^3.

I've discovered the Universe will last at least another 3^^^3 years

This implies that physical properties like the cosmological constant and the half-life of protons can be measured to a precision of roughly 1/3^^^3 relative error.

To me it seems like both of those claims have prior probability ~ 1/3^^^3. (How many spaceships would you have to build and how long would you have to test them to get an MTTF estimate as large as 3^^^3? How many measurements do you have to make to get the standard deviation below 1/3^^^3?)

Why can't you increase your asymptote with new evidence? If, for instance, your utility was bounded at 2^160 utilons before the mugger opened the sky then just increase your bound according to that evidence and then shut up and multiply to decide whether to pay $5. You can't update to a bound of 3^^^3 in one step since you can't receive enough evidence at once, which is a handy feature for avoiding muggings, but your utility at a distant point in the future is essentially unbounded given enough evidential updates over time.

Useful utility bounds should be derivable from our knowledge of the universe. If we can theoretically create 10^80 unique, just-worth-living lives with the estimated matter and energy in the universe then that provides a minimum bound, although it's probably desirable to choose the bound large enough that the 10^80th life is worth nearly as much as the 1st or 10^11th life. When we have evidence for a change in our estimate of the available matter and energy or a change in the efficiency of turning matter and energy into utility we scale the bound appropriately.

Which hypothesis, between "2*3" and "6", is simpler and less complex, based on what we observe from these two different machines? Which one is right? AFAIK, this is still completely unresolved.

If we're considering hypotheses across all mathematically possible universes then why not consider hypotheses across all mathematically possible languages/machines as well?

However, Alice is very bright, and is the type of person who can adapt herself to many situations and learn skills quickly. If Alice were to spend the first six months of college deeply immersing herself in studying business, she would probably start developing a passion for business. If she purposefully exposed herself to certain pro-business memeplexes (e.g. watched a movie glamorizing the life of Wall Street bankers), then she could speed up this process even further. After a few years of taking business classes, she would probably begin to forget what about English literature was so appealing to her, and be extremely grateful that she made the decision she did. Therefore she would gain the same 2 mu from having a job she is passionate about, along with an additional 1 mu from being rich, meaning that the 3 mu choice of business wins out over the 2 mu choice of English.

Isn't it better to find ways to be happy given one's current characteristics than to change those characteristics? Maybe Alice should get a business degree and make enough money to spend time studying English literature for 3 + 2 = 5 mu, or even in the worst case 3+epsilon mu if she absolutely hates business and never appreciates it enough to gain more than epsilon utility from having the degree other than the 1 mu from the better job with which to fund an English degree.

Maybe I'm misreading your post and you didn't intend to suggest that Alice completely gives up something which has genuine utility in favor of altering herself to acquire utility from something that previously didn't yield any.

"Right, this is the real core of Pascal's Mugging [...]. For aggregative utility functions over a model of the environment which e.g. treat all sentient beings (or all paperclips) as having equal value without diminishing marginal returns, and all epistemic models which induce simplicity-weighted explanations of sensory experience, all decisions will be dominated by tiny variances in the probability of extremely unlikely hypotheses because the "model size" of a hypothesis can grow Busy-Beaver faster than its Kolmogorov complexity."

I think others have expressed doubts that the promised utility of hypotheses should be able to grow in Busy-Beaver fashion faster than a properly updated probability, but I don't see a similar argument on your list. It seems to have been taken for granted because a simple hypothesis clearly can describe a very large utility.

But what is the probability that any process, including a Matrix Lord, can successfully perform O(3^^^3) computations/operations/actions? Even a Matrix Lord should have a mean time to failure (MTTF) and the estimate of MTTF should be directly derivable from the complexity of the Matrix Lord. A Matrix Lord of complexity O(2^256) should have a MTTF like O(2^256) operations, not O(3^^^3) operations. Even if each operation produces 1 utilon without regard to whether the process completes or not that limits the expected number of utilons is 1*MTTF or O(2^256), making the expected value O(2^256) * P(Matrix Lord will is telling the truth | Matrix Lord of complexity 2^256 exists).

This doesn't work if the MTTF increases over time due perhaps to a self-improving and self-sustaining process, but that actually makes sense. I'm far more likely to believe a mugger that says "Give me $5 or I'm going to release a self-replicating nanobot that will turn the universe into grey goo" or "Give me $5 or I turn on this UFAI" than one that says "Give me $5 or I'll switch off the Matrix that's running your universe." "Give me $5 or I'll start an unreasonably long process that eventually will produce so much negative utility that it overrides all your probability estimation ability" just makes me think that whatever process the mugger may have will fail long before the expected value of accepting would become positive.

The limit as predicted MTTF goes to infinity is probably just Pascal's Wager. If you believe that an entity can keep a Matrix running infinitely long, that entity can affect your utility infinitely. That probably requires an (eventually) infinitely complex entity so that it can continue even if any finite number of its components fail. That leaves the door open to being mugged by a self-improving process in an infinite universe, but I think that also makes sense. Bound your utility if you're worried about infinite utility.

Given that, what does it mean when I say "I think we should express 'action A increases the value of utility function U' in math as X", which seems like a sensible statement?

I think it makes sense as a statement about decision theories. How would a choice of which mathematical expression of 'action A increases the value of utility function U' affect actual utility? Only by affecting which actions are chosen; in other words by selecting a particular (class of) decision theory which maximizes utility due in part to its expression of what "should" means mathematically.