Ah - I was waiting for the first commenter to draw the analogy with Counterfactual Mugging. The problem is, Psy-Kosh's scenario does not contain any predictors, amnesia, copying, simulations or other weird stuff that we usually use to break decision theories. So it's unclear why standard decision theory fails here.
So it's unclear why standard decision theory fails here.
To the first approximation, clearly, because you destructively update and thus stop caring about the counterfactuals. Shouldn't do that. The remaining questions are all about how the standard updating works at all, and in what situations that can be used, and so by extension why it can't be used here.
The source is here. I'll restate the problem in simpler terms:
You are one of a group of 10 people who care about saving African kids. You will all be put in separate rooms, then I will flip a coin. If the coin comes up heads, a random one of you will be designated as the "decider". If it comes up tails, nine of you will be designated as "deciders". Next, I will tell everyone their status, without telling the status of others. Each decider will be asked to say "yea" or "nay". If the coin came up tails and all nine deciders say "yea", I donate $1000 to VillageReach. If the coin came up heads and the sole decider says "yea", I donate only $100. If all deciders say "nay", I donate $700 regardless of the result of the coin toss. If the deciders disagree, I don't donate anything.
First let's work out what joint strategy you should coordinate on beforehand. If everyone pledges to answer "yea" in case they end up as deciders, you get 0.5*1000 + 0.5*100 = 550 expected donation. Pledging to say "nay" gives 700 for sure, so it's the better strategy.
But consider what happens when you're already in your room, and I tell you that you're a decider, and you don't know how many other deciders there are. This gives you new information you didn't know before - no anthropic funny business, just your regular kind of information - so you should do a Bayesian update: the coin is 90% likely to have come up tails. So saying "yea" gives 0.9*1000 + 0.1*100 = 910 expected donation. This looks more attractive than the 700 for "nay", so you decide to go with "yea" after all.
Only one answer can be correct. Which is it and why?
(No points for saying that UDT or reflective consistency forces the first solution. If that's your answer, you must also find the error in the second one.)