I get that. What I'm really wondering is how this extends to probabilistic reasoning. I can think of an obvious analog. If the algorithm assigns zero probability that it will choose $5, then when it explores the counterfactual hypothesis "I choose $5", it gets nonsense when it tries to condition on the hypothesis. That is, for all U,
is undefined. But is there an analog for this problem under uncertainty, or was my sketch correct about how that would work out?
A causal reasoner will compute about P(utility=U| do{action=$5}), which doesn't run into this trouble. This is the approach I recommend.
Probabilistic reasoning about actions that you will make is, to the best of my knowledge, not a seriously considered approach to making decisions outside of the context of mixed strategies in game theory, and even there it doesn't apply that strong, as you can see mixed strategies as putting forth a certain (but parameterized) action whose outcome is subject to uncertainty.
I don't think your sketch is correct for two reaso...
If it's worth saying, but not worth its own post, even in Discussion, it goes here.