Perhaps I should have been more specific than to use a vague term like "morality". Replace it with CEV, since that should be the sum total of all your values.

Most people value happiness, so let me use that as an example. Even if I value own happiness 1000x more than other people's happiness, if there are more than 1000 people in the word, then the vast majority of my concern for happiness is still external to myself. One could do this same calculation for all other values, and add them up to get CEV, which is likely to be weighted toward others for the same reason that happiness is.

Of course, perhaps some people legitimately would prefer 3^^^3 dust specs in people's eyes to their own death. And perhaps some people's values aren't coherent, such as preferring A to B, B to C, and C to A. But if neither of these is the case, then replacing one's self with a more efficient agent maximizing the same values should be a net gain in most cases.