It reminds me of a thought experiment I have read somewhere. Imagine that there are many people in the world who are dying from starvation. They would happily agree to be your slaves, if you feed them. There is too many of them and they are not qualified for modern economy; if you would give them more than a minimum, there wouldn't be enough for you to have a decent life. Imagine you only have the following three options:

A) Share everything with them. Everyone will live, but everyone will be rather poor.
B) Accept them as your slaves, in exchange for food and shelter. Everyone will live, you will keep your quality of life, but there will be a huge inequality.
C) Refuse to interact with them. You will keep your quality of life, but they will die from starvation.

If we order these options by altruism, which is how those poor people would see them, we get A > B > C. It would be best to make those poor people our equals, but even helping them survive as slaves is better than letting them die.

If we order these options by pure egoism, we get B > C > A. Having slaves would be a cool improvement, keeping status quo is acceptable.

But in the typical decision process, we refuse B to signal that we are not complete egoists, and refuse A because we are not really that much altruistic. Thus what remains is the option C... which paradoxically both altruists and egoists consider to be worse than B (and the altruists also worse than A).