The problem is how classical logical statements work. The statement "If A then B" more properly translates as "~(A and ~B)".
Thus, we get valid logical statements that look bizarre to humans: "If Paris is the capital of France, then Rome is the capital of Italy" seems untrue in a causal sense (if we changed the capital of France, we would not change the capital of Italy, and vice versa) but it is true in a logical sense, because A is true, B is true, true and ~true is false, and ~false is true.
That example seems just silly, but the problem is the reverse example is disastrous. Notice that, because of the "and," if A is false then it doesn't matter what B is: false and X is false, ~false is true. If I choose the premise "Marseilles is the capital of France," then any B works. "If Marseilles is the capital of France, then I will receive infinite utility" is a true relationship under classical logic, but is clearly not a causal relationship: changing the capital will not grant me infinite utility, and as soon as the capital changes, the logical truth of the sentence will change.
If you have a reasoner that makes decisions, they need to use causal logic, not classical logic, or they'll get tripped up by the word "implication."
I get that. What I'm really wondering is how this extends to probabilistic reasoning. I can think of an obvious analog. If the algorithm assigns zero probability that it will choose $5, then when it explores the counterfactual hypothesis "I choose $5", it gets nonsense when it tries to condition on the hypothesis. That is, for all U,
is undefined. But is there an analog for this problem under uncertainty, or was my sketch correct about how that would work out?
If it's worth saying, but not worth its own post, even in Discussion, it goes here.