I’ve noticed that the Axiom of Independence does not seem to make sense when dealing with indexical uncertainty, which suggests that Expected Utility Theory may not apply in situations involving indexical uncertainty. But Googling for "indexical uncertainty" in combination with either "independence axiom" or “axiom of independence” give zero results, so either I’m the first person to notice this, I’m missing something, or I’m not using the right search terms. Maybe the LessWrong community can help me figure out which is the case.
The Axiom of Independence says that for any A, B, C, and p, you prefer A to B if and only if you prefer p A + (1-p) C to p B + (1-p) C. This makes sense if p is a probability about the state of the world. (In the following, I'll use “state” and “possible world” interchangeably.) In that case, what it’s saying is that what you prefer (e.g., A to B) in one possible world shouldn’t be affected by what occurs (C) in other possible worlds. Why should it, if only one possible world is actual?
In Expected Utility Theory, for each choice (i.e. option) you have, you iterate over the possible states of the world, compute the utility of the consequences of that choice given that state, then combine the separately computed utilities into an expected utility for that choice. The Axiom of Independence is what makes it possible to compute the utility of a choice in one state independently of its consequences in other states.
But what if p represents an indexical uncertainty, which is uncertainty about where (or when) you are in the world? In that case, what occurs at one location in the world can easily interact with what occurs at another location, either physically, or in one’s preferences. If there is physical interaction, then “consequences of a choice at a location” is ill-defined. If there is preferential interaction, then “utility of the consequences of a choice at a location” is ill-defined. In either case, it doesn’t seem possible to compute the utility of the consequences of a choice at each location separately and then combine them into a probability-weighted average.
Here’s another way to think about this. In the expression “p A + (1-p) C” that’s part of the Axiom of Independence, p was originally supposed to be the probability of a possible world being actual and A denotes the consequences of a choice in that possible world. We could say that A is local with respect to p. What happens if p is an indexical probability instead? Since there are no sharp boundaries between locations in a world, we can’t redefine A to be local with respect to p. And if A still denotes the global consequences of a choice in a possible world, then “p A + (1-p) C” would mean two different sets of global consequences in the same world, which is nonsensical.
If I’m right, the notion of a “probability of being at a location” will have to acquire an instrumental meaning in an extended decision theory. Until then, it’s not completely clear what people are really arguing about when they argue about such probabilities, for example in papers about the Simulation Argument and the Sleeping Beauty Problem.
Edit: Here's a game that exhibits what I call "preferential interaction" between locations. You are copied in your sleep, and both of you wake up in identical rooms with 3 buttons. Button A immunizes you with vaccine A, button B immunizes you with vaccine B. Button C has the effect of A if you're the original, and the effect of B if you're the clone. Your goal is to make sure at least one of you is immunized with an effective vaccine, so you press C.
To analyze this decision in Expected Utility Theory, we have to specify the consequences of each choice at each location. If we let these be local consequences, so that pressing A has the consequence "immunizes me with vaccine A", then what I prefer at each location depends on what happens at the other location. If my counterpart is vaccinated with A, then I'd prefer to be vaccinated with B, and vice versa. "immunizes me with vaccine A" by itself can't be assigned an utility.
What if we use the global consequences instead, so that pressing A has the consequence "immunizes both of us with vaccine A"? Then a choice's consequences do not differ by location, and “probability of being at a location” no longer has a role to play in the decision.
This example does not really illustrate the point, but I think I see where you are going.
Suppose there is room with two buttons X, and Y. Pushing button X gives you $100 (Event A) with probability p, and does nothing (Event C) with probability 1-p, every time it is pushed. Pushing button Y gives you $150 (Event B) with the same probability p, and does nothing (Event C) with probability 1-p, provided that Event B has not yet occurred, otherwise it does nothing (Event C).
So, now you get to play a game, where you enter the room and get to press either button X or Y, and then your memory is erased, you are reintroduced to the game, and you get to enter the room again (indistinguishable to you from entering the first time), and press either button X or Y.
Because of indexical uncertainty, you have to make the same decision both times (unless you have a source of randomness). So, your expected return from pressing X is 2*p*$100 (the sum from two independent events with expected return p*$100), and your expected return from pressing Y is (1-(1-p)^2) * $150 (the payoff times the probability of not failing to get the payoff two times), which simplifies to (2*p - p^2) * $150.
So, difference in the payoffs, P(Y) - P(X) = 2*p * $50 - (p^2) * $150 = $50 * p * (2 - 3*p). So Y is favored for values of p between 0 and 2/3, and X is favored for values of p between 2/3 and 1.
But doesn't the Axiom of Independence say that Y should be favored for all values of p, because Event B is preferred to Event A? No, because pressing Y does not really give p*B + (1-p)*C. It gives q*p*B + (1-q*p)*C, where q is the probability that Event B has not already happened. Given that you press Y two times, and you do not know which time is which, q = (1 - .5 * p), that is, the probability that it is not the case that this is the second time (.5), and the B happened the first time (p). Now, if I had chosen different probabilities for the behaviors of the buttons, so that when factoring in the indexical uncertainty, the resulting probabilities in the game were equal, then the Axiom of Independence would apply.
Your analysis looks correct to me. But if Wei Dai indeed meant something like your example, why did he/she say "indexical uncertainty" instead of "amnesia"? Can anyone provide an example without amnesia - a game where each player gets instantiated only once - showing the same problems? Or do people that say "indexical uncertainty" always imply "amnesia"?