Reading the post you linked to, it feels like some sort of fallacy is at work in the thought experiment as the results are tallied up.
Specifically: suppose we live in copies-matter world, and furthermore suppose we create a multiverse of 100 copies, 90 of which get the good outcome and 10 of which get the bad outcome (using the aforementioned biased quantum coin, which through sheer luck gives us an exact 90:10 split across 100 uncorrelated flips). Since copies matter, we can conclude it's a moral good to post hoc shut down 9 of the 10 bad-outcome copies and replace those simulacra with 9 duplicates of existing good-outcome copies. While we've done a moral wrong by discontinuing 9 bad-outcome copies, we do a greater moral right by creating 9 new good-outcome copies, and thus we paperclip-maximize our way toward greater net utility.
Moreover, still living in copies-matter world, it's a net win to shut down the final bad-outcome copy (i.e. "murder", for lack of a better term, the last of the bad-outcome copies) and replace that final copy with one more good-outcome copy, thus guaranteeing that the outcome for all copies is good with 100% odds. Even supposing the delta between the good outcome and the bad outcome was merely one speck of dust in the eye, and furthermore supposing that the final bad-outcome copy was content with the bad outcome and would have preferred to continue existing.
At this point, the overall multiverse outcome is identical to the quantum coin having double heads, so we might as well have not involved quantum pocket change in the first place. Instead, knowing that one outcome was better than the other, we should have just forced the known-good outcome on all copies in the first place. With that, copies-matter world and copies-don't-matter world are now reunified.
Returning to copies-don't-matter world (and our intuition that that's where we live), it feels like there's an almost-but-not-quite-obvious analogy with Shannon entropy and/or Kolmogorov-Chaitin complexity lurking just under the surface.
Ruminating further, I think I've narrowed down the region where the fallacious step occurs.
Suppose there are 100 simulacra, and suppose for each simulacrum you flip a coin biased 9:1 in favor of heads. You choose one of two actions for each simulacrum, depending on whether the coin shows heads or tails, but the two actions have equal net utility for the simulacra so there are no moral conundra. Now, even though the combination of 90 heads and 10 tails is the most common, the permutations comprising it are nonetheless vastly outnumbered by all the remaini...
Future technologies pose a number of challenges to moral philosophy. One that I think has been largely neglected is the status of independent identical copies. (By "independent identical copies" I mean copies of a mind that do not physically influence each other, but haven't diverged because they are deterministic and have the same algorithms and inputs.) To illustrate what I mean, consider the following thought experiment. Suppose Omega appears to you and says:
You and all other humans have been living in a simulation. There are 100 identical copies of the simulation distributed across the real universe, and I'm appearing to all of you simultaneously. The copies do not communicate with each other, but all started with the same deterministic code and data, and due to the extremely high reliability of the computing substrate they're running on, have kept in sync with each other and will with near certainty do so until the end of the universe. But now the organization that is responsible for maintaining the simulation servers has nearly run out of money. They're faced with 2 possible choices:
A. Shut down all but one copy of the simulation. That copy will be maintained until the universe ends, but the 99 other copies will instantly disintegrate into dust.
B. Enter into a fair gamble at 99:1 odds with their remaining money. If they win, they can use the winnings to keep all of the servers running. But if they lose, they have to shut down all copies.
According to that organization's ethical guidelines (a version of utilitarianism), they are indifferent between the two choices and were just going to pick one randomly. But I have interceded on your behalf, and am letting you make this choice instead.
Personally, I would not be indifferent between these choices. I would prefer A to B, and I guess that most people would do so as well.
I prefer A because of what might be called "identical copy immortality" (in analogy with quantum immortality). This intuition says that extra identical copies of me don't add much utility, and destroying some of them, as long as one copy lives on, doesn't reduce much utility. Besides this thought experiment, identical copy immortality is also evident in the low value we see in the "tiling" scenario, in which a (misguided) AI fills the the universe with identical copies of some mind that it thinks is optimal, for example one that is experiencing great pleasure.
Why is this a problem? Because it's not clear how it fits in with the various ethical systems that have been proposed. For example, utilitarianism says that each individual should be valued independently of others, and then added together to form an aggregate value. This seems to imply that each additional copy should receive full, undiscounted value, in conflict with the intuition of identical copy immortality.
Similar issues arise in various forms of ethical egoism. In hedonism, for example, does doubling the number of identical copies of oneself double the value of pleasure one experiences, or not? Why?
A full ethical account of independent identical copies would have to address the questions of quantum immortality and "modal immortality" (cf. modal realism), which I think are both special cases of identical copy immortality. In short, independent identical copies of us exist in other quantum branches, and in other possible worlds, so identical copy immortality seems to imply that we shouldn't care much about dying, as long as some copies of us live on in those other "places". Clearly, our intuition of identical copy immortality does not extend fully to quantum branches, and even less to other possible worlds, but we don't seem to have a theory of why that should be the case.
A full account should also address more complex cases, such as when the copies are not fully independent, or not fully identical.
I'm raising the problem here without having a good idea how to solve it. In fact, some of my own ideas seem to conflict with this intuition in a way that I don't know how to resolve. So if anyone has a suggestion, or pointers to existing work that I may have missed, I look forward to your comments.