Hi Holden,

I just read this thread today. I made a clarification upthread about the description of my comment above, under Louie's. Also, I'd like to register that I thought your characterization of that interview as such was fine, even without the clarifications you make here.

They both argue that standard Bayesian inference indicates against the literal use of non-robust expected value estimates, particularly in "Pascal's Mugging" type scenarios.

As a technical point, I don't think these posts address "Pascal's Mugging" scenarios in any meaningful way.

Bayesian adjustment is a standard part of Pascal's Mugging. The problem is that Solomonoff complexity priors have fat tails, because describing fundamental laws of physics that allow large payoffs is not radically more complex than laws that only allow small payoffs. It doesn't take an extra 10^1000 bits to describe a world where an action generates 2^(10^1000) times as many, e.g. happy puppies. So we can't rule out black swans a priori in that framework (without something like an anthropic assumption that amounts to the Doomsday Argument).

The only thing in your posts that could help with Pascal's Mugging is the assumption of infinite certainty in a distribution without relevantly fat tails or black swans, like a normal or log-normal distribution. But that would be an extreme move, taking coherent worlds of equal simplicity and massively penalizing the ones with high payoffs, so that no evidence that could fit in a human brain could convince us we were in the high-payoff worlds. Without some justification, that seems to amount to assuming the problem away, not addressing it.

Disclaimer 1: This is about expected value measured in the currency of "goods" like happy puppies, rather than expected utility, since agents can have bounded utility, e.g. simply not caring much more about saving a billion billion puppies rather than a billion. This seems fairly true of most people, at least emotionally.

Disclaimer 2: Occam's razor priors give high value to Pascal's Mugging cases, but they also give higher expectations to all other actions. For instance, the chance that space colonization will let huge populations be created increases the expected value of reducing existential risk by many orders of magnitude to total utilitarians. But it also greatly increases the expected payoffs of anything else that reduces existential risk by even a little. So if vaccinating African kids is expected to improve the odds of human survival going forward (not obvious but plausible) then its expected value will be driven to within sight of focused existential risk reductions, e.g. vaccination might be a billionth the cost-effectiveness of focused risk-reduction efforts but probably not smaller by a factor of 10^20. By the same token, different focused existential risk interventions will compete against one another, so one will not want to support the relatively ineffective ones.

Thanks for the helpful comments! I was uninformed about all those details above.

These posts are not about GiveWell's process.

One of the posts has the sub-heading "The GiveWell approach" and all of the analysis in both posts use examples of charities you're comparing. I agree you weren't just talking about the GiveWell process... you were talking about a larger philosophy of science you have that informs things like the GiveWell process.

I recognize that you're making sophisticated arguments for your points. Especially the assumptions that you claim simply must be true to satisfy your intuition that charities should be rewarded for transparency and punished otherwise. Those seem wise from a "getting things done" point of view for an org like GiveWell -- even when there is no mathematical reason those assumptions should be true -- but only a human-level tit-for-tat shame/enforcement mechanism you hope eventually makes this circularly "true" through repeated application. Seems fair enough.

But adding regression adjustments to cancel out the effectiveness of any charity which looks too effective to be believed (based on the common sense of the evaluator) seems like a pretty big finger on the scale. Why do so much analysis in the beginning if the last step of the algorithm is just "re-adjust effectiveness and expected value to equal what feels right"? Your adjustment factor amounts to a kind of Egalitarian Effectiveness Assumption: We are all created equal at turning money into goodness. Or perhaps it's more of a negative statement, like, "None of us is any better than the best of us at turning money into goodness" -- where the upper limit on the best is something like 1000x or whatever the evaluator has encountered in the past. Any claims made above the best limit gets adjusted back down -- those guys were trying to Pascal's Mug us! That's the way in which there's a blinding effect. You disbelieve the claims of any groups who claims to be more effective per capita than you think is possible.

A few corrections.

• I know that Holden interviewed two other supporters of ours... but I don't think he interviewed 2 other employees. If he did, why did he only publish the unhelpful notes from the one employee he spoke to who didn't know anything?

• SIAI didn't give Jasen to GiveWell to be interviewed -- Holden chose him unilaterally -- not because he was a good choice, but because Jasen is from New York (just like Holden).

• I'm unaware of Holden sending his notes to anyone at SIAI prior to publication. Who did he send them to? I never saw them.

• My guess is Holden sent his notes back to Jasen and called that "sending them to SIAI for feedback". In other words, no one at SIAI who is a leader, or a board member, or someone who understands the plans/finances of the organization saw the notes prior to publication. If Holden had sent the notes to any of the board members of Singularity Institute, they would have sent him tons of corrections.

• To clarify, I didn't say the interview itself was a lie. I said calling it an interview with SIAI was a lie. I stick by that characterization.

A few quick notes:

• As I wrote in my response to Carl on The GiveWell Blog, the conceptual content of this post does not rely on the assumption that the value of donations (as measured in something like "lives saved" or "DALYs saved") is normally distributed. In particular, a lognormal distribution fits easily into the above framework. .

• I recognize that my model doesn't perfectly describe reality, especially for edge cases. However, I think it is more sophisticated than any model I know of that contradicts its big-picture conceptual conclusions (e.g., by implying "the higher your back-of-the-envelope [extremely error-prone] expected-value calculation, the necessarily higher your posterior expected-value estimate") and that further sophistication would likely leave the big-picture conceptual conclusions in place.

• JGWeissman is correct that I meant "maximum" when I said "inflection point."

Hello all,

Thanks for the thoughtful comments. Without responding to all threads, I'd like to address a few of the themes that came up. FYI, there are also interesting discussions of this post at The GiveWell Blog , Overcoming Bias , and Quomodocumque (the latter includes Terence Tao's thoughts on "Pascal's Mugging").

On what I'm arguing. There seems to be confusion on which of the following I am arguing:

(1) The conceptual idea of maximizing expected value is problematic.

(2) Explicit estimates of expected value are problematic and can't be taken literally.

(3) Explicit estimates of expected value are problematic/can't be taken literally when they don't include a Bayesian adjustment of the kind outlined in my post.

As several have noted, I do not argue (1). I do aim to give with the aim of maximizing expected good accomplished, and in particular I consider myself risk-neutral in giving.

I strongly endorse (3) and there doesn't seem to be disagreement on this point.

I endorse (2) as well, though less strongly than I endorse (3). I am open to the idea of formally performing a Bayesian adjustment, and if this formalization is well done enough, taking the adjusted expected-value estimate literally. However,

• I have examined a lot of expected-value estimates relevant to giving, including those done by the DCP2 , Copenhagen Consensus , and Poverty Action Lab , and have never once seen a formalized adjustment of this kind.

• I believe that often - particularly in the domains discussed here - formalizing such an adjustment in a reasonable way is simply not feasible and that using intuition is superior. This is argued briefly in this post, and Dario Amodei and Jonah Sinick have an excellent exchange further exploring this idea at the GiveWell Blog.

• If you disagree with the above point, and feel that such adjustments ought to be done formally, then you do disagree with a substantial part of my post; however, you ought to find the remainder of the post more consequential than I do, since it implies substantial room for improvement in the most prominent cost-effectiveness estimates (and perhaps all cost-effectiveness estimates) in the domains under discussion.

All of the above applies to expected-value calculations that take relatively large amounts of guesswork, such as in the domain of giving. There are expected-value estimates that I feel are precise/robust enough to take literally.

Is it reasonable to model existential risk reduction and/or "Pascal's Mugging" using log-/normal distributions? Several have pointed out that existential risk reduction and "Pascal's Mugging" seem to involve "either-or" scenarios that aren't well approximated by log-/normal distributions. I wish to emphasize that I'm focused on the prior over expected value of one's actions and on the distribution of error in one's expected-value estimate. (The latter is a fuzzy concept that may be best formalized with the aid of concepts such as imprecise probability. In the scenarios under discussion, one often must estimate the probability of catastrophe essentially by making a wild guess with a wide confidence interval, leaving wide room for "estimate error" around the expected-value calculation.) Bayesian adjustments to expected-value estimates of actions, in this framework, are smaller (all else equal) for well-modeled and well-understood "either-or" scenarios than for poorly-modeled and poorly-understood "either-or" scenarios.

For both the prior and for the "estimate error," I think the log-/normal distribution can be a reasonable approximation, especially when considering the uncertainty around the impact of one's actions on the probability of catastrophe.

The basic framework of this post still applies, and many of its conclusions may as well, even when other types of probability distributions are assumed.

My views on existential risk reduction are outside the scope of this post. The only mention I make of existential risk reduction is to critique the argument that "charities working on reducing the risk of sudden human extinction must be the best ones to support, since the value of saving the human race is so high that 'any imaginable probability of success' would lead to a higher expected value for these charities than for others." Note that Eliezer Yudkowsky and Michael Vassar also appear to disapprove of this argument, so it seems clear that disputing this argument is not the same as arguing against existential risk reduction charities.

For the past few years we have considered catastrophic risk reduction charities to be lower on GiveWell's priority list for investigation than developing-world aid charities, but still relatively high on the list in the scheme of things. I've recently started investigating these causes a bit more, starting with SIAI (see LW posts on my discussion with SIAI representatives and my exchange with Jaan Tallinn). It's plausible to me that asteroid risk reduction is a promising area, but I haven't looked into it enough (yet) to comment more on that.

My informal objections to what I term EEV. Several have criticized the section of my post giving informal objections to what I term the EEV approach (by which I meant explicitly estimating expected value using a rough calculation and not performing a Bayesian adjustment). This section was intended only as a very rough sketch of what unnerves me about EEV; there doesn't seem to be much dispute over the more formal argument I made against EEV; thus, I don't plan on responding to critiques of this section.

Completely agree with your general point on marginal analysis (although I'm a TDT skeptic), and am a fan of GiveWell, but this is trivially wrong:

It is not possible for everyone to behave this way in elections: no voter is able to consider the existing distribution of votes before casting their own.

This seems to assume away information about the size of the electorate as well as any predictive power about the outcome. Surely the marginal benefit of a Presidential vote in a small swing state is massively higher than in a large solidly Democratic state, for example. And in addition to historical results, there is polling data in advance of the election to improve predictions.

Besides this being theoretically true, we can see it empirically from the spending patterns of both Presidential campaigns and political parties on Congressional races. They allocate money to the states / races where they believe it will do the most marginal good, which is often a very inequal distribution. Thus they do, in fact "consider the existing distribution of votes before casting" their advertising dollars.

I take it that you're suggesting marginal analysis based on the standard correct classical causal decision theory (in which no one is responsible for saving a life by donating blood unless someone would have actually died had that donation not been made) out of either belated humility about the probability of an SIAI-originating decision theory being correct, or because you're planning to actually convince someone and you don't want to invoke Hofstadterian superrationality in place of the standard correct decision theory?

:)

My guess would be that at the margin, a blood donation saves less than 0.00001 lives. (Otherwise, compensation would be increased for the paid donors). But, if you want to use a TDT/UDT style analysis, here are some relevant statistics from the American Red Cross:

• The number of blood donations collected in the U.S. in a year: 16 million (2006).
• The number of patients who receive blood in the U.S. in a year: 5 million (2006).

Given these numbers, I would estimate that roughly 0.5 million (US) lives are saved (more accurately, extended) by blood products annually. If you adopt the assumption that all blood comes from voluntary, uncompensated donations, and divide those 0.5 million lives among the 16 million annual donations, you get one life saved for every 32 pints donated - not as much as jsteinhardt hoped, but still significant enough to earn a major warm-and-fuzzy.

It is possible that Donor A may choose to donate fully to GiveWell for many reasons, including a prior assumption that it's 50:50 or better without checking easily available facts. This reflects badly on Donor A, not GiveWell, and does not in any way make a case for calling GiveWell "ethically questionable". The most you could possibly say is "GiveWell does not overly pander to the lowest common denominator enough" but these people are already donating their money to Make a Wish foundation or something equally silly.

I belabour this point because charities run solely on their appearance as ethical, and to the extent that your comments deprive GiveWell of possible donations on the basis of spurious claims, you're doing a bad thing.