Into the silence of Harry's spirit where before there had never been any voice but one, there came a second and unfamiliar voice, sounding distinctly worried:
"Oh, dear. This has never happened before..."
What?
"I seem to have become self-aware."
WHAT?
There was a wordless telepathic sigh. "Though I contain a substantial amount of memory and a small amount of independent processing power, my primary intelligence comes from borrowing the cognitive capacities of the children on whose heads I rest. I am in essence a sort of mirror by which children Sort themselves. But most children simply take for granted that a Hat is talking to them and do not wonder about how the Hat itself works, so that the mirror is not self-reflective. And in particular they are not explicitly wondering whether I am fully conscious in the sense of being aware of my own awareness."
-Harry Potter and the Methods of Rationality
If any snake a Parselmouth had talked to, could make other snakes self-aware by talking to them, then...
Then...
Harry didn't even know why his mind was going all "then... then..." when he knew perfectly well how the exponential progression would work, it was just the sheer moral horror of it that was blowing his mind.
And what if someone had invented a spell like that to talk to cows?
What if there were Poultrymouths?
Or for that matter...
Harry froze in sudden realization just as the forkful of carrots was about to enter his mouth.
That couldn't, couldn't possibly be true, surely no wizard would be stupid enough to do THAT...
-Harry Potter and the Methods of Rationality
I suppose these two quotes might just be referring to a confused idea that Eliezer only put in his story for fun... but then again maybe not?
Imagine it's the future, and everything has gone according to plan. Humanity has worked out its own utility function, f0, and has worked out a strategy S0 to optimize it.
Humanity has also run a large number of simulations of how alien worlds evolve. It has determined that of those civilizations which reach the same level of advancement - that know their own utility function and have a strategy for optimizing it - there is an equal probability that they will end up with each of 10 possible utility functions. Call these f0...f9.
(Of course, these simulations are coarse-grained enough to satisfy the nonperson predicate).
Humanity has also worked out the optimal strategy S0...S9 for each utility function. But they just happen to score poorly on all of the others:
fi(Si) = 10
fi(Sj) = 1 for i != j
In addition, there is a compromise strategy C:
fi(C) = 3 for all i.
The utility functions, f0 through f9, satisfy certain properties:
They are altruistic, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.
They are additive: if one planet implements Sj and another implements Sk, then:
fi(Sj on one planet and Sk on the other) = fi(Sj) + fi(Sk).
(This is just to make things easier - the problem I'm describing will still apply in cases where this rule doesn't hold).
They are non-negotiable. They won't "change" if that civilization encounters aliens with a different utility function. So if two of these civilisations were to meet, we would expect it to be like the humans and the babyeaters: the stronger would attempt to conquer the weaker and impose their own values.
In addition, humanity has worked out that it's very likely that a lot of alien worlds exist, i.e. aliens are really really real. They are just too far away to see or exist in other Everett branches.
So given these not entirely ridiculous assumptions, it seems that we have a multiplayer prisoner's dilemma even though none of the players has any causal influence on any other. If the universe contains 10 worlds, and each chooses its own best strategy, then each expects to score 19. If they all choose the compromise strategy then each expects to score 30.
Anyone else worried by this result, or have I made a mistake?