I have trouble understanding how both of these can be true. If situations where it applies are very infrequent, how essential can it really be?
What I should have said is "When discussing or thinking about morality we consider situations where it applies very infrequently." When people think about morality, and posit moral dilemmas, they typically only consider situations where everyone involved is capable of interacting. When people consider the Trolley Problem they only consider the six people on the tracks and the one person with the switch.
I suppose that technically separability applies to every decision we make. For every action we take there is a possibility that someone, somewhere does not approve of our taking it and would stop us if they could. This is especially true if the universe is as vast as we now think it is. So we need separability in order to discount the desires of those extremely causally distant people.
To avoid paralysis, utilitarians need some way of resolving intersubjective differences in utility calculation for the same shared world-state. Using "separability" to discount the unknowable utility calculations of unknown Matrioshka Brains is a negligible portion of the work that needs to be done here.
You are certainly right that separability isn't the only thing that utilitarianism needs to avoid paralysis, and that there are other issues that it needs to resolve before it even gets to the stage where separability is needed. I'm merely saying that, at that particular stage, separability is essential. It certainly isn't the only possible way utilitarianism could be paralyzed, or otherwise run into problems.
For my own part, I would spend considerably more than a cent to create an identical copy of myself whom I can interact with
When I refer to identical copies I mean a copy that starts out identical to me, and remains identical throughout its entire lifespan, like the copies that exist in parallel universes, or the ones in this matrix-scenario Wei Dai describes. You appear to also be using "identical" to refer to copies that start out identical, but diverge later and have different experiences.
Like you, I would probably pay to create copies I could interact with, but I'm not sure how enthusiastic about it I would be. This is because I find experiences to be much more valuable if I can remember them afterwards and compare them to other experiences. If both mes get net value out of the experience like you expect then this isn't a relevant concern. But I certainly wouldn't consider having 3650 copies of me existing for one day and then being deleted to be equivalent to living an extra 10 years the way Robin Hanson appears to.
Followup to: Poll: What value extra copies?
For those of you who didn't follow Eliezer's Quantum Physics Sequence, let me reiterate that there is something very messed up about the universe we live in. Specifically, the Many Worlds Interpretation (MWI) of quantum mechanics states that our entire classical world gets copied something like 1040±20 times per second1. You are not a line through time, but a branching tree.
If you think carefully about Descartes' "I think therefore I am" type skepticism, and approach your stream of sensory observations from such a skeptical point of view, you should note that if you really were just one branch-line in a person-tree, it would feel exactly the same as if you were a unique person-line through time, because looking backwards, a tree looks like a line, and your memory can only look backwards.
However, the rules of quantum mechanics mean that the integral of the modulus squared of the amplitude density, ∫|Ψ|2, is conserved in the copying process. Therefore, the tree that is you has branches that get thinner (where thickness is ∫|Ψ|2 over the localized density "blob" that represents that branch) as they branch off. In fact they get thinner in such a way that if you gathered them together into a bundle, the bundle would be as thick as the trunk it came from.
Now, since each copying event creates a slightly different classical universe, the copies in each of the sub-branches will each experience random events going differently. This means that over a timescale of decades, they will be totally "different" people, with different jobs, probably different partners and will live in different places though they will (of course) have your DNA, approximate physical appearance, and an identical history up until the time they branched off. For timescales on the order of a day, I suspect that almost all of the copies will be virtually identical to you, even down to going to bed at the same time, having exactly the same schedule that day, thinking almost all of the same thoughts etc.
MWI mixes copies and probability
When a "random" event happens, either the event was pseudorandom (like a large digit of pi) or it was a copy event, meaning that both (or all) outcomes were realized elsewhere in the wavefunction. This means that in many situations, when you say "there is a probability p of event X happening", what this really means is "proportion p of my copy-children will experience X".
LW doesn't care about copies
In Poll: What value extra copies?, I asked what value people placed upon non-interacting extra copies of themselves, asking both about lock-step identical and statistically identical copies. The overwhelming opinion was that neither were of much value. For example, Sly comments:2
"I would place 0 value on a copy that does not interact with me. This might be odd, but a copy of me that is non-interacting is indistinguishable from a copy of someone else that is non-interacting. Why does it matter that it is a copy of me?"
How to get away with attempted murder
Suppose you throw a grenade with a quantum detonator at Sly. The detonator will sample a qbit in an even superposition of states 1 and 0. On a 0 it explodes, instantly vaporizing sly (it's a very powerful grenade). On a 1, it defuses the grenade and dispenses a $100 dollar note. Suppose that you throw it and observe that it doesn't explode:
(A) does Sly charge you with attempted murder, or does he thank you for giving him $100 in exchange for something that had no value to him anyway?
(B) if he thanks you for the free $100, does he ask for another one of those nice free hundred dollar note dispensers? (This is the "quantum suicide" option
(C) if he says "the one you've already given me was great, but no more please", then presumably if you throw another one against his will, he will thank you for the free $100 again. And so on ad infinitum. Sly is temporally inconsistent if this option is chosen.
The punch line is that the physics we run on gives us a very strong reason to care about the welfare of copies of ourselves, which is (according to my survey) a counter-intuitive result.
EDIT: Quite a few people are biting the quantum suicide bullet. I think I'll have to talk about that next. Also, Wei Dai summarizes:
Another way to think about this is that many of us seem to share the follow three intuitions about non-interacting extra copies, out of which we have to give up at least one to retain logical consistency:
I might add a fourth option that many people in the comments seem to be going after: (4) We don't intrinsically value copies in other branches, we just have a subjective anticipation of becoming them.
1: The copying events are not discrete, rather they consist of a continuous deformation of probability amplitude in state space, but the shape of that deformation looks a lot like a continuous approximation to a discrete copying event, and the classical rules of physics approximately govern the time evolution of the "copies" as if they were completely independent. This last statement is the phenomenon of decoherence. The uncertainty in the copying rate is due to my ignorance, and I would welcome a physicist correcting me.
2: There were many others who expressed roughly similar views, and I don't hold it as a "black mark" to pick the option that I am advising against, rather I encourage people to honestly put forward their opinions in a spirit of communal learning.