Many people on Less Wrong see Judea Pearl's work as the right approach to formalizing causality and getting the right conditional probabilities. But if it was simple to formally specify the right causal model of the world given sensory information, we'd probably have AGI already.

re: threats: I don't see how exactly a brain based AI can be even seen as a 'threat' to survival of humanity. I instead see it as about the only way that permits humanity to survive at all. First of all, the simbrain is a part of humanity itself precisely in the way in which from-scratch AI isn't, second, even if the simbrains are a little bit charitable towards the original humans, that's your FAI.

Also:

when you don't know if you have lesion and the probability of having lesion.

You would still have priors for all of these things.

edit: ahh, wait, the EDT is some pretty naive theory that can not even process anything as complicated as evidence for causality working in our universe. Whatever then, a thoughtless approach leads to thoughtless results, end of story. The correct decision theory should be able to control for pre-existing lesion when it makes sense to do so.

I think you've got it. Pure EDT and CDT really just are that stupid - and irredeemably so because agents implementing them will not want to learn how to replace their decision strategy (beyond resolving themselves to their respective predetermined stable outcomes). Usually when people think either of them are a good idea it is because they have been incidentally supplementing and subverting them with a whole lot of their own common sense!

edit: ahh, wait, the EDT is some pretty naive theory that can not even process anything as complicated as evidence for causality working in our universe.

Can you explain this?

EDT is described as $V(A) = \sum_{j} P(O_j | A) U(O_j)$. If you have knowledge about the mechanisms behind the how the lesion causes smoking, that would change $P(A | O_j)$ and therefore also $P(O_j | A)$.

I would like to ask for help on how to use expected utility maximization, in practice, to maximally achieve my goals.

I think the best single-sentence answer is: don't.

(Quick comment before I go offline for today.)

Here is the problem. If I use expected utility maximization (EU) on big and unintuitive problems like existential risks and to decide what I should do about it; If I use EU to decide how to organize my life by and large; If I use EU to decide to pursue a terminal goal but then stop using it to decide what goals are instrumental in achieving the desired outcome, then how does it help to use EU at all? And otherwise, how do I decide where to draw the line?

People closely associated with SIAI/LW do use EU in support of their overall goals, yet ignore EU when it comes to flying to NY or writing a book about rationality:

[S]uppose you have a moral view that counts future people as being worth as much as present people. You might say that fundamentally it doesn't matter whether someone exists at the current time or at some future time, just as many people think that from a fundamental moral point of view, it doesn't matter where somebody is spatially---somebody isn't automatically worth less because you move them to the moon or to Africa or something. A human life is a human life. If you have that moral point of view that future generations matter in proportion to their population numbers, then you get this very stark implication that existential risk mitigation has a much higher utility than pretty much anything else that you could do.

-- Nick Bostrom

If you want to maximize your marginal expected utility you have to maximize on your choice of problem over the combination of high impact, high variance, possible points of leverage, and few other people working on it. The problem of stable goal systems in self-improving Artificial Intelligence has no realistic competitors under any three of these criteria, let alone all four.

-- Eliezer Yudkowsky

In terms of expected utility maximization, even large probabilities of jumping the interval between a universe-history in which 95% of existing biological species survive Earth’s 21st century, versus a universe-history where 80% of species survive, are just about impossible to trade off against tiny probabilities of jumping the interval between interesting universe-histories, versus boring ones where intelligent life goes extinct, or the wrong sort of AI self-improves....with millions of people working on environmentalism, and major existential risks that are completely ignored… if you add a marginal resource that can, rarely, be steered by expected utilities instead of warm glows, devoting that resource to environmentalism does not make sense.

-- Eliezer Yudkowsky

That part is correct, but opting not to smoke for the purpose of avoiding this increase in probability s an error.

I still don't see how it is. If the agent has no other information, all he knows is that if he decides to smoke it is more likely that he has the lesion. His decision itself doesn't influence whether he has the lesion, of course. But he desires to not have the lesion, and therefore should desire to decide not to smoke.

The way the lesion influences deciding to smoke will be through the utility function or the decision theory. With no other information, the agent can't trust that his decision will outsmart the lesion.

Some thinking on what to think about is very important, unfortunately it is also very hard to get it right. For example here we can discuss optimal decisions involving probability, entirely forgetting the limited runtime and the effect of introducing risk on the efficacy of bounded calculations in the future. When you take risks, for example, you double the size of the expected-utility-calculating tree, meaning that in limited time you cut down on depth.

Then there's this: you can think how to optimize your behaviour by single digit percentage, for example, by trying to do nonbiased estimates of your utility, which you won't be doing very well anyway because the world is hard to predict. Or you can spend that runtime e.g. learning to program, then coming up and writing some popular application, putting it up on a relevant store, and getting way more than enough money for papering over your inefficiencies.

re: pill.

The important thing is that you should expect, with very good confidence, to have found the toxic effects of the drug if there were any. If it is so, then not having found such effects is good evidence. You do not expect to have a proof that ghosts do not exist, if they don't, that's what makes 'because' be a fallacy. You do not expect to have a proof the pills are unsafe, before you did proper testing, either; and even after testing it may easily be unsafe and there's certain risk remaining. The reasoning about pills used to be every bit as fallacious as the reasoning about ghosts - and far more deadly, before the mid 20th century or so, from which point we arranged the testing as to expect to find most of the toxic effects before approving drugs.

re: circularity

Well, if there weren't any other claims about electrons or god, those two claims would not even be claims but simply word definitions. The 'entity we think we seen everywhere and call electrons leaves tracks because of such and such' is the real argument, and 'god the creator of the universe personally wrote the bible' is the real argument. If we actually expected bible to be less likely to exist without God, then the bible would have been evidence, but as such I'd say the likehood of bible-like-religious-text is at very best entirely unrelated to existence or non-existence of god.

That's btw the way my atheism is, except other way around: nothing in religion is coupled in any way what so ever to existence or non-existence of god; i don't need to know if god exists or not to entirely reject religion and be strongly anti-religious. If anything, existence of multiple religions and the evil things that religions did and how they clashed with each other, would seem like a kind of thing that would be less likely in universe that has the creator god who watches over it, and constitute a (weak) evidence against existence of god (and fairly strong evidence against existence of god that intervenes etc). For me the existence of religions (the way they are) is a weak evidence that God does not exist.

I agree with you. I don't think that EDT is wrong on the Smoking Lesion. Suppose that, in the world of this problem, you see someone else decide to smoke. What do you conclude from that? Your posterior probability of that person having the lesion goes up over the prior. Now what if that person is you? I think the same logic should apply.