The following is an honest non-rhetorical question: Is it not misleading to use the word 'cooperation' as you seem to be using it here? Don't you still get 'cooperation' in this sense if the subsets of agents are not causally interacting with each other (say) but have still semi-Platonicly 'merged' via implicit logical interaction as compared to some wider context of decision algorithms that by logical necessity exhibit comparatively less merging? This sets up a situation where an agent can (even accidentally) engineer 'Pareto improvements' just by improving its decision algorithm (or more precisely replacing 'its' decision algorithm (everywhere 'it' is instantiated, of course...) with a new one that has the relevant properties of a new, possibly very different logical reference class). It's a total bastardization of the concept of trade but it seems to be enough to result in some acausal economy (er, that is, some positive-affect-laden mysterious timeless attractor simultaneously constructed and instantiated by timeful interaction) or 'global cooperation' as you put it, and yet despite all that timeless interaction there are many ways it could turn out that would not look to our flawed timeful minds like cooperation. I don't trust my intuitions about what 'cooperation' would look like at levels of organization or intelligence much different from my own, so I'm hesitant to use the word.
(I realize this is 'debating definitions' but connotations matter a lot when everything is so fuzzily abstract and yet somewhat affect-laden, I think. And anyway I'm not sure I'm actually debating definitions because I might be missing an important property of Pareto improvements that makes their application to agents that are logical-property-shifting-over-time not only a useless analogy but a confused one.)
This question is partially prompted by your post about the use of the word 'blackmail' as if it was technically clear and not just intuitively clear which interactions are blackmail, trade, cooperation, et cetera, outside of human social perception (which is of course probably correlated with more-objectively-correct-than-modern-human meta-ethical truths but definitely not precisely so).
If the above still looks like word salad to you... sigh please let me know so I can avoid pestering you 'til I've worked more on making my concepts and sentences clearer. (If it still looks way too much like word salad but you at least get the gist, that'd be good to know too.)
Is it not misleading to use the word 'cooperation' as you seem to be using it here?
Yes, it's better to just say that there is probably some acausal morally relevant interaction, wherein the agents work on their own goals.
(I don't understand what you were saying about time/causality. I disagree with Nesov_2009's treatment of preference as magical substance inherent in parts of things.)
I’ve noticed that the Axiom of Independence does not seem to make sense when dealing with indexical uncertainty, which suggests that Expected Utility Theory may not apply in situations involving indexical uncertainty. But Googling for "indexical uncertainty" in combination with either "independence axiom" or “axiom of independence” give zero results, so either I’m the first person to notice this, I’m missing something, or I’m not using the right search terms. Maybe the LessWrong community can help me figure out which is the case.
The Axiom of Independence says that for any A, B, C, and p, you prefer A to B if and only if you prefer p A + (1-p) C to p B + (1-p) C. This makes sense if p is a probability about the state of the world. (In the following, I'll use “state” and “possible world” interchangeably.) In that case, what it’s saying is that what you prefer (e.g., A to B) in one possible world shouldn’t be affected by what occurs (C) in other possible worlds. Why should it, if only one possible world is actual?
In Expected Utility Theory, for each choice (i.e. option) you have, you iterate over the possible states of the world, compute the utility of the consequences of that choice given that state, then combine the separately computed utilities into an expected utility for that choice. The Axiom of Independence is what makes it possible to compute the utility of a choice in one state independently of its consequences in other states.
But what if p represents an indexical uncertainty, which is uncertainty about where (or when) you are in the world? In that case, what occurs at one location in the world can easily interact with what occurs at another location, either physically, or in one’s preferences. If there is physical interaction, then “consequences of a choice at a location” is ill-defined. If there is preferential interaction, then “utility of the consequences of a choice at a location” is ill-defined. In either case, it doesn’t seem possible to compute the utility of the consequences of a choice at each location separately and then combine them into a probability-weighted average.
Here’s another way to think about this. In the expression “p A + (1-p) C” that’s part of the Axiom of Independence, p was originally supposed to be the probability of a possible world being actual and A denotes the consequences of a choice in that possible world. We could say that A is local with respect to p. What happens if p is an indexical probability instead? Since there are no sharp boundaries between locations in a world, we can’t redefine A to be local with respect to p. And if A still denotes the global consequences of a choice in a possible world, then “p A + (1-p) C” would mean two different sets of global consequences in the same world, which is nonsensical.
If I’m right, the notion of a “probability of being at a location” will have to acquire an instrumental meaning in an extended decision theory. Until then, it’s not completely clear what people are really arguing about when they argue about such probabilities, for example in papers about the Simulation Argument and the Sleeping Beauty Problem.
Edit: Here's a game that exhibits what I call "preferential interaction" between locations. You are copied in your sleep, and both of you wake up in identical rooms with 3 buttons. Button A immunizes you with vaccine A, button B immunizes you with vaccine B. Button C has the effect of A if you're the original, and the effect of B if you're the clone. Your goal is to make sure at least one of you is immunized with an effective vaccine, so you press C.
To analyze this decision in Expected Utility Theory, we have to specify the consequences of each choice at each location. If we let these be local consequences, so that pressing A has the consequence "immunizes me with vaccine A", then what I prefer at each location depends on what happens at the other location. If my counterpart is vaccinated with A, then I'd prefer to be vaccinated with B, and vice versa. "immunizes me with vaccine A" by itself can't be assigned an utility.
What if we use the global consequences instead, so that pressing A has the consequence "immunizes both of us with vaccine A"? Then a choice's consequences do not differ by location, and “probability of being at a location” no longer has a role to play in the decision.