Babyeater's dilemma

Giles

Imagine it's the future, and everything has gone according to plan. Humanity has worked out its own utility function, f₀, and has worked out a strategy S₀ to optimize it.

Humanity has also run a large number of simulations of how alien worlds evolve. It has determined that of those civilizations which reach the same level of advancement - that know their own utility function and have a strategy for optimizing it - there is an equal probability that they will end up with each of 10 possible utility functions. Call these f₀...f₉.

(Of course, these simulations are coarse-grained enough to satisfy the nonperson predicate).

Humanity has also worked out the optimal strategy S₀...S₉ for each utility function. But they just happen to score poorly on all of the others:

f_i(S_i) = 10
f_i(S_j) = 1 for i != j

In addition, there is a compromise strategy C:

f_i(C) = 3 for all i.

The utility functions, f₀ through f₉, satisfy certain properties:

They are altruistic, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.

They are additive: if one planet implements S_j and another implements S_k, then:
f_i(S_j on one planet and S_k on the other) = f_i(S_j) + f_i(S_k).

(This is just to make things easier - the problem I'm describing will still apply in cases where this rule doesn't hold).

They are non-negotiable. They won't "change" if that civilization encounters aliens with a different utility function. So if two of these civilisations were to meet, we would expect it to be like the humans and the babyeaters: the stronger would attempt to conquer the weaker and impose their own values.

In addition, humanity has worked out that it's very likely that a lot of alien worlds exist, i.e. aliens are really really real. They are just too far away to see or exist in other Everett branches.

So given these not entirely ridiculous assumptions, it seems that we have a multiplayer prisoner's dilemma even though none of the players has any causal influence on any other. If the universe contains 10 worlds, and each chooses its own best strategy, then each expects to score 19. If they all choose the compromise strategy then each expects to score 30.

Anyone else worried by this result, or have I made a mistake?

They are altruistic, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.

They are additive: if one planet implements S_j and another implements S_k, then:
f_i(S_j on one planet and S_k on the other) = f_i(S_j) + f_i(S_k).

(This is just to make things easier - the problem I'm describing will still apply in cases where this rule doesn't hold).

Anyone else worried by this result, or have I made a mistake?

To update my reply to Silas Barta, after a little reflection I would say this:

The various species are supposed to possess common knowledge of each other's utility functions, and of each other's epistemic beliefs about how these utility functions can be satsified.

Since the various species' preferences are described by utility functions, we must assume that each species has self-modified collectively (or so the humans believe) such that they collectively obey the von Neumann-Morgenstern axioms - this eliminates much of the complexity that I had in mind when I wrote my reply to Barta.

However, one further item of common knowledge would be helpful: common knowledge of whether the various species are likely to be timeless decision theorists. If they possess this common knowledge, then the dilemma is just a disguised version of a standard Newcomblike problem: the agents possess common knowledge of all the relevant factors that might influence the abstract computation that they implement in determining which strategy to employ. This is no different to the scenario in which two AIs can read one another's source code - except in this case they do it by magic (the scenario is entirely ridiculous, I'm afraid). And in that case co-operation in the prisoner's dilemma is the optimal choice.

On the other hand if they don't possess common knowledge of whether they are TDT-agents (rather than CDT-agents for example) then whether it is wise for humans to defect or co-operate depends on their probability estimate regarding whether the aliens are mostly TDT-agents, and their estimates of the aliens' own estimates whether the other species are TDT-agents, et cetera. I don't really know how that infinite regress would be resolved by the humans, and your premises give us little way of knowing what these probability estimates might be.