Thanks to Stuart for pointing me to this.
I think this example is an interesting illustration of how difficult it is to "solve the academic coordination problem," that is to be up to speed on what folks in related disciplines are up to.
Folks who actually worry about exposures like smoking (epidemiologists) are not only aware of this issue, they invented a special target of inference which addresses it. This target is called "effect of treatment on the treated," (ETT) and in potential outcome notation you would write it like so:
E[C(s) | s] - E[C(s') | s]
where C(s) is cancer rate under the smoking intervention, C(s') is cancer rate under the non-smoking intervention, and in both cases we are also conditioning on whether the person is a natural smoker or not.
By the consistency axiom, E[C(s) | s] = E[C | s], but the second term is harder. In fact, the second term is very subtle to identify and sometimes may require untestable assumptions. Note that ETT is causal. EDT has no way of talking about this quantity. I believe I mentioned ETT at MIRI before, in another context.
In fact, not only have epidemiologists invented ETT, but they are usually more interested in ETT than the ACE (average causal effect, which is the more common effect you read about on wikipedia). In other words, an epidemiologist will not optimise utility with respect to p(Outcome(action)), but with respect to p(Outcome(action) | how you naturally act), precisely because how you naturally act gives information about how interventions affect outcomes.
The reason CDT fails on this example is because it is "leaving info on the table," not because representing causal relationships is the wrong thing to do. The reason EDT succeeds on this example is because despite the fact that EDT is not representing the situation correctly, the numbers in the problem happen to be benign enough that the bias from not taking confounding into account properly is "cancelled out."
The correct repair for CDT is to not leave info on the table. The correct repair for EDT, as always, is to represent confounding and thus become causal. This is also the lesson for Newcomb problems: CDT needs to properly represent the problem, and EDT needs to start representing confounding properly -- it is trivial to modify Newcomb's problem to have confounding which will cause EDT to give the wrong answer. That was the point of my FHI talk.
I stumbled upon this paper by Andy Egan and thought that its main result should be shared. We have the Newcomb problem as counterexample to CDT, but that can be dismissed as being speculative or science-fictiony. In this paper, Andy Egan constructs a smoking lesion counterexample to CDT, and makes the fascinating claim that one can construct counterexamples to CDT by starting from any counterexample to EDT and modifying it systematically.
The "smoking lesion" counterexample to EDT goes like this:
EDT implies that she should not smoke (since the likely outcome in a world where she doesn't smoke is better than the likely outcome in a world where she does). CDT correctly allows her to smoke: she shouldn't care about the information revealed by her preferences.
But we can modify this problem to become a counterexample to CDT, as follows:
Here EDT correctly tells her not to smoke. CDT refuses to use her possible decision as evidence that she has the gene and tells her to smoke. But this makes her very likely to get cancer, as she is very likely to have the gene given that she smokes.
The idea behind this new example is that EDT runs into paradoxes whenever there is a common cause (G) of both some action (S) and some undesirable consequence (C). We then take that problem and modify it so that there is a common cause G of both some action (S) and of a causal relationship between that action and the undesirable consequence (S→C). This is then often a paradox of CDT.
It isn't perfect match - for instance if the gene G were common, then CDT would say not to smoke in the modified smoker's lesion. But it still seems that most EDT paradoxes can be adapted to become paradoxes of CDT.