If you have uncertainty, this doesn't apply anymore

I am not sure I understand. Uncertainty in what? Plus, if you are going beyond the VNM Theorem, what is the utility function we're talking about, anyway?

If you have uncertainty, this doesn't apply anymore

I am not sure I understand. Uncertainty in what? Plus, if you are going beyond the VNM Theorem, what is the utility function we're talking about, anyway?

I am not sure I understand. Uncertainty in what?

In the outcome of each action. If the world is deterministic, then all that matters is a preference ranking over outcomes. This is called ordinal utility.

If the outcomes for each action are sampled from some action-dependent probability distribution, then a simple ranking isn't enough to express your preferences. VNM theory allows you to specify a cardinal utility function, which is invariant only up to positive affine transform.

In practice this is needed to model common human preferences like risk-aversion w.r.t. money.

Probability is a bounded variable.

Solomonoff induction combined with an unbounded utility function gives undefined expectations. But Solomonoff induction combined with a bounded utility function can give defined expectations.

And Solomonoff induction by itself gives defined predictions.

Solomonoff induction combined with an unbounded utility function gives undefined expectations. But Solomonoff induction combined with a bounded utility function can give defined expectations.

Yes.

And Solomonoff induction by itself gives defined predictions.

If you try to use it to estimate the expectation of any unbounded variable, you get an undefined value.

But it looks like the shape of the distributions *isn't* normal-like? In fact, that's one of the standard EA arguments for why it's important to spend energy on finding the most effective thing you can do: if possible intervention outcomes really *were* approximately normally distributed, then your exact choice of an intervention wouldn't matter all that much. But actually the distribution of outcomes looks very skewed; to quote The moral imperative towards cost-effectiveness:

DCP2 includes cost-effectiveness estimates for 108 health interventions, which are presented in the chart below, arranged from least effective to most effective [...] This larger sample of interventions is even more disparate in terms of costeffectiveness. The least effective intervention analysed is still the treatment for Kaposi’s sarcoma, but there are also interventions up to ten times more cost-effective than education for high risk groups. In total, the interventions are spread over more than four orders of magnitude, ranging from 0.02 to 300 DALYs per $1,000, with a median of 5. Thus, moving money from the least effective intervention to the most effective would produce about 15,000 times the benefit, and even moving it from the median intervention to the most effective would produce about 60 times the benefit.

It can also be seen that due to the skewed distribution, the most effective interventions produce a disproportionate amount of the benefits. According to the DCP2 data, if we funded all of these interventions equally, 80% of the benefits would be produced by the top 20% of the interventions. [...]

Moreover, there have been health interventions that are even more effective than any of those studied in the DCP2. [...] For instance in the case of smallpox, the total cost of eradication was about $400 million. Since more than 100 million lives have been saved so far, this has come to less than $4 per life saved — significantly superior to all interventions in the DCP2.

I think you misunderstood what I said or I didn't explain myself well: I'm not assuming that the DALY distribution obtained if you choose interventions at random is normal. I'm assuming that for each intervention, the DALY distribution it produces is normal, with an intervention-dependent mean and variance.

I think that for the kind of interventions that GiveWell considers, this is a reasonable assumption: if the number of DALYs produced by each intervention is the result of a sum of many roughly independent variables (e.g. DALYs gained by helping Alice, DALYs gained by helping Bob, etc.) the total should be approximately normally distributed, due to the central limit theorem.

For other types of interventions, e.g. whether to fund a research project, you may want to use a more general family of distributions that allows non-zero skewness (e.g. skew-normal distributions), but as long as the distribution is light-tailed and you don't use extreme values for the parameters, you would not run into Pascal's Mugging issues.

EY can imagine all the fictional scenario he wants, this doesn't mean that we should assign non-negligible probabilities to them.

Not negligible, zero. You literally can not believe in an theory of physics that allows large amounts of computing power. If we discover that an existing theory like quantum physics allows us to create large computers, we will be forced to abandon it.

If your epistemic model generates undefined expectations when you combine it with your utility function, then I'm pretty sure we can say that at least one of them is broken.

Yes something is broken, but it's definitely not our prior probabilities. Something like solomonoff induction should generate perfectly sensible predictions about the world. If knowing those predictions makes you do weird things, that's a problem with your decision procedure. Not the probability function.

Not negligible, zero.

You seem to have a problem with very small probabilities but not with very large numbers. I've also noticed this in Scott Alexander and others. If very small probabilities are zeros, then very large numbers are infinities.

You literally can not believe in an theory of physics that allows large amounts of computing power. If we discover that an existing theory like quantum physics allows us to create large computers, we will be forced to abandon it.

Sure. But since we know no such theory, there is no a priori reason to assume it exists with non-negligible probability.

Something like Solomonoff induction should generate perfectly sensible predictions about the world.

Nope, it doesn't. If you apply Solomonoff induction to predict arbitrary integers, you get undefined expectations.

Cryonics organization should encourage independent replication instead or playing the victim.

Independent would mean research for which they aren't paying. How should Alcor go about encouraging such research in your opinion?

There's actually been some cool studies on DNA extracted from brain tissue from people born before the first nuclear tests...

DNA is turned over for the most part during DNA replication. Neurons are terminally differentiated and do not divide, and thus their DNA is more or less frozen in place even as other molecules turn over (barring small repair events). People born before the first nuclear tests have neurons bearing a different carbon isotope ratio in their DNA than people born afterwards, and this was used in a study to determine the rate over a human lifetime of new neurogenesis versus nerve cells that stick with you for your whole life. Turns out most neurons stick around but in particular regions like your hippocampus there is a good deal of turnover of cells, with only about two thirds of the DNA there with you from birth:

http://www.physicscentral.com/explore/action/images/Neuron-by-age.jpg

The turnover of DNA molecules doesn't really make any difference, just something fun that lets you track cells incidentally.

Nice information!

There seems to be no obvious reason to assume that the probability falls exactly in proportion to the number of lives saved.

If GiveWell told me they thought that real-life intervention A could save one life with probability PA and real-life intervention B could save a hundred lives with probability PB, I'm pretty sure that dividing PB by 100 would be the wrong move to make.

There seems to be no obvious reason to assume that the probability falls exactly in proportion to the number of lives saved.

It is an assumption to make asymptotically (that is, for the tails of the distribution), which is reasonable due to all the nice properties of exponential family distributions.

If GiveWell told me they thought that real-life intervention A could save one life with probability PA and real-life intervention B could save a hundred lives with probability PB, I'm pretty sure that dividing PB by 100 would be the wrong move to make.

I'm not implying that.

EDIT:

As a simple example, if you model the number of lives saved by each intervention as a normal distribution, you are immune to Pascal's Muggings. In fact, if your utility is linear in the number of lives saved, you'll just need to compare the means of these distributions and take the maximum. Black swan events at the tails don't affect your decision process.

Using normal distributions may be perhaps appropriate when evaluating GiveWell interventions, but for a general purpose decision process you will have, for each action, a probability distribution over possible future world state trajectories, which when combined with an utility function, will yield a generally complicated and multimodal distribution over utility. But as long as the shape of the distribution at the tails is normal-like, you wouldn't be affected by Pascal's Muggings.

View more: Next

Yes, you need risk tolerance / risk preference as well, but once we have that, aren't we already outside of the VNM universe?

No, risk tolerance / risk preference can be modeled with VNM theory.