Let's look at a slightly more complicated example:

A patient comes in to the hospital because he's sick, his vitals are taken (C), and based on C, a doctor prescribes medication A. Sometime later, the patient dies (Y). Say this happens for lots of patients, and we form an empirical distribution p(C,A,Y) from these cases. Things we may want to represent:

marginal probability of death: p(Y)

the policy the doctor followed when prescribing the medicine: p(A|C)

the probability that someone would die given that they were given the medicine: p(Y|A)

the "causal effect" of medicine on death: ???

The issue with the "causal effect" is that the doctor's policy, if it is sensible, will be more likely to prescribe A to people who are already very sick, and not prescribe A to people who are mostly healthy. Thus it may very well turn out that p(Y|A) is higher than the probability p(Y) (this happens for example with HIV drugs). But surely this doesn't mean that medicine doesn't help! What we need is the probability of death given that the doctor arbitrarily decided to give medicine, regardless of background status C. This arbitrary decision "decouples" the influence of the patient's health status and the influence of the medicine in the sense that if we average over health status for patients after such a decision was made, we would get just the effect of the medicine itself.

For this we need to distinguish an "arbitrary" decision to give medicine, divorced from the policy. Call this arbitrary decision A* . You may then claim that we can just use rules of conditional probabilities on events Y,C,A, and A* , and we do not need to involve anything else. Of course, if you try to write down sensible axioms that relate A* with Y,C,A you will notice that essentially you are writing down standard axioms of potential outcomes (one way of thinking about potential outcomes is they are random variables after a hypothetical "arbitrary decision" like A* ). For example, if were to arbitrarily decide to prescribe medicine just precisely for the patients the doctor prescribed medicine to, we would get that p(Y|A) = p(Y|A* ) (this is known as the consistency axiom). You will find that your A* functions as an intervention do(a).

There is a big literature on how intervention events differ from observation events (they have different axiomatizations, for example). You may choose to use a notation for interventions that does not look different from observations, but the underlying math will be different if you wish to be sensible. That is the point. You need new math beyond math about evidence (which is what probability theory is).

That seems a little bit ad hoc to me. Either you care about dogs (and then even the tiniest non-zero amount of caring should be enough for the argument) or you don't.

Eliezer handled this sort of objection in Newcomb's Problem and Regret of Rationality:

You can't tell me, first, that above all I must conform to a particular ritual of cognition, and then that, if I conform to that ritual, I must change my morality to avoid being Dutch-booked. Toss out the losing ritual; don't change the definition of winning.

What you are doing here is insisting that I conform to your ritual of cognition (i.e. total utilitarianism with real-number valuation and additive aggregation). I see no reason to accede to such a demand.

The following are facts about what I do and don't care about:

1) All else being equal, I prefer that a dog not be tortured.
2) All else being equal, I prefer that my grandmother not be tortured.
3) I prefer any number of dogs being tortured to my grandmother being tortured.
4 through ∞) Some other stuff about my preferences, skipped for brevity.

#s 2 and 3 are very strong preferences. #1 is less so.

Now I want to find a moral calculus that captures those facts. You, on the other hand, are telling me that, first, I must accept your moral calculus, and then, that if I do so, I must toss out one of the aforementioned preferences.

I decline to do either of those things. (As Eliezer says in the above link: The utility function is not up for grabs.)

But for the sake of the argument : if a hyperintelligent alien came to the Earth (eg. an AI), what would you propose how the alien should figure out which mechanisms in the universe should be of moral concern?

I don't know. This is the kind of thing that demonstrates why we need FAI theory and CEV.

What would you think of the agent’s morality if it discounted your welfare lexically?

I would think that its morality is different from mine. Also, I would be sad, because presumably such a morality on the AI's part would result in bad things for me. Your point?

By the way, the dogs vs. grandma case differs in an important way from specks vs. torture:

The specks are happening to humans.

It is not actually inconsistent to choose TORTURE in specks/torture while choosing GRANDMA in dogs/grandma. All you have to do is value humans (and humans' utility) while not valuing dogs (or placing dogs on a "lower moral tier" than your grandmother/humans in general).

In other words, "do many specks add up to torture" and "do many dogs add up to grandma" are not the same question.

Sure. However, you raise what is in principle a very solid objection, and so I would like to address it.

Let's say that I would, all else being equal, prefer that a dog not be tortured. Perhaps I am even willing to take certain actions to prevent a dog from being tortured. Perhaps I also think that two dogs being tortured is worse than one dog being tortured, etc.

However, I am willing to let that dog, or a million dogs, or any number of dogs, be tortured to save my grandmother from the same fate.

What are we to make of this?

In that case, some component of our utilitarianism might have to be re-examined. Perhaps dogs have a nonzero value, and a lot of dogs have more value than only a few dogs, but no quantity of dogs adds up to one grandmother; but on the other hand, some things are worth more than one grandmother (two grandmothers? all of humanity?).

Real numbers do not behave this way. Perhaps they are not a sufficient number system for utilitarian calculations.

(Of course, it's possible to suppose that we could, if we chose, construct various hypotheticals (perhaps involving some complex series of bets) which would tease out some inconsistency in that set of valuations. That may be the case here, but nothing obvious jumps out at me.)

Hello everyone!

I'm a 21 years old and study medicine plus bayesian statistics and economics. I've been lurking LW for about half a year and I now feel sufficiently updated to participate actively. I highly appreciate this high-quality gathering of clear-thinkers working towards a sane world. Therefore I oftenly pass LW posts on to guys with promising predictors in order to shorten their inferential distance. I'm interested in fixing science, bayesian reasoning, future scenarios (how likely is dystopia, i.e. astronomical amounts of suffering?), machine intelligence, game theory, decision theory, reductionism (e.g. of personal identity), population ethics and cognitive psychology. Thanks for all the lottery winnings so far!