Thus, for you, hearing about the problem, there is a 0.5 probability of your encountering the problem as stated, and a 0.5 probability of your encountering the corresponding situation, in which Omega either hands you $10000 or doesn't, based on his prediction. This is all very fine and rational.

Eliezer wrote:

It seems like I want to decide "as if" I don't know whether the coin came up heads or tails, and then implement that decision even if I know the coin came up heads. But I don't have a good formal way of talking about how my decision in one state of knowledge has to be determined by the decision I would make if I occupied a different epistemic state, conditioning using the probability previously possessed by events I have since learned the outcome of... Again, it's easy to talk informally about why you have to reply "Yes" in this case, but that's not the same as being able to exhibit a general algorithm.

Your post seems more appropriate as a comment to Eliezer's post. Your example with the fluctuating probabilities just shows that you didn't arrive at your "fine and rational" solution by computing with a generalized decision theory. You just guess-and-checked the two possible decisions to find the reflectively consistent one.

So Eliezer has asked: What mathematical formalism should a rational agent use to represent decision problems that crop up in its environment?

A causal decision theorist would tell you that the agent can use a Markov decision process. But in counterfactual-mugging-like situations, an MDP doesn't define a quantity that a reflectively self-consistent agent would maximize.

The challenge is to present a formalism in which to represent decision problems that might include some level of "decision-dependent counterfactual outcomes", and define what quantity is to be maximized for each formalized problem-instance.