As far as I can reconstruct EDT's algorithm, it goes something like this:
1) I know that smoking is correlated with lung cancer.
2) I've read in a medical journal that smoking and lung cancer have a common cause, some kind of genetic lesion. I don't know if I have that lesion.
3) I'd like to smoke now, but I'm not sure if that's the best decision.
4) My friend, a causal decision theorist, told me that smoking or not smoking cannot affect the lesion that I already have or don't. But I don't completely buy that reasoning. I prefer to use something else, which I will call "evidential decision theory".
5) To figure out the best action to take, first I will counterfactually imagine myself as an automaton whose actions are chosen randomly, taking into account the lesion that I have or don't, using the frequencies observed in the world. So an automaton with the lesion will have a higher probability of smoking and a higher probability of cancer.
6) Next, I will figure out what the automaton's actions say about its utility, using ordinary conditional probabilities and expected values. It looks like the utility of automatons that smoke is lower than the utility of those that don't, because the former ones are more likely to get cancer.
7) Now I will remember that I'm not an automaton, and choose to avoid smoking based on the above reasoning!
Does that make sense?
The problem with this line of reasoning is that the desire to smoke is correlated with smoking, and therefore with the genetic lesion. Since and EDT agent is assumed to perform Bayesian updates, it should update its probability of having the lesion upon the observation that it has a desire to smoke.
How much it should update depends on its prior.
If, according to its prior, the desire to smoke largely screens off the correlation between the lesion and smoking, then the agent will choose to smoke.
ErinFlight said:
Thinking about it, I realized that this might be a common concern. There are probably plenty of people who've looked at various more-or-less technical or jargony Less Wrong posts, tried understanding them, and then given up (without posting a comment explaining their confusion).
So I figured that it might be good to have a thread where you can ask for explanations for any Less Wrong post that you didn't understand and would like to, but don't want to directly comment on for any reason (e.g. because you're feeling embarassed, because the post is too old to attract much traffic, etc.). In the spirit of various Stupid Questions threads, you're explicitly encouraged to ask even for the kinds of explanations that you feel you "should" get even yourself, or where you feel like you could get it if you just put in the effort (but then never did).
You can ask to have some specific confusing term or analogy explained, or to get the main content of a post briefly summarized in plain English and without jargon, or anything else. (Of course, there are some posts that simply cannot be explained in non-technical terms, such as the ones in the Quantum Mechanics sequence.) And of course, you're encouraged to provide explanations to others!