Standard cost-benefit analysis on non-Venusian global warming involves (implicit or explicit) projections of climate sensitivity, technological change, economic and population growth, risks of nuclear war and other global catastrophic risks, economic damages of climate change, and more 90 (!!!) years into the future or even centuries. There are huge areas where subjective estimates play big roles there.
Right, so maybe reference to global warming was a bad example because there too, one is dealing with vast uncertainties. Note that global warming passes the "test" (3) above.
Nonetheless, one can put reasonable probability distributions on these and conclude that there are low-hanging fruit worth plucking (as part of a big enough global x-risk reduction fund).
I'm curious about this.
Regarding social costs and being an "odd duck": note that Weitzman, in his widely celebrated article on uncertainty about seemingly implausible hard-to-analyze high-impact climate change, also calls for work on risks of AI and engineered pathogens as some of the handful of serious x-risks demanding attention.
Likewise judge-economist Richard Posner called for preliminary work on AI extinction risk in his book Catastrophe. Philosopher John Leslie in his book on human extinction discussed AI risk at length. Bill Gates went out of his way to mention it as a possibility.
These are pertinent examples but I think it's still fair to say that interest in reducing AI risks marks one as an odd duck at present and that this gives rise to an equilibrating force against successful work on preventing AI extinction risk (how large I don't know). I can imagine this changing in the near future.
Regarding Fermi calculations, the specific argument in the post is wrong for the reasons JGWeissman mentions:
I attempted to clarify in the comments.
With respect to sacrifice/demandingness, that's pretty orthogonal to efficacy
What I was trying to get at here was that in part for social reasons and in part because of the inherent openendedness/uncertainty spanning over many orders of magnitude being conducive to psychological instability, the minimum sacrifice needed to usefully consciously reduce existential risk may be too great for people to effectively work to reduce existential risk by design.
Regarding Pascal's Mugging, that involves numbers much more extreme than show up in the area of x-risk, by many orders of magnitude.
This is true for the Eliezer/Bostrom case study, but my intuition is that the same considerations apply. Even if the best estimates that the probability that Christianity is true aren't presently > 10^(-50), there was some point in the past when in Europe the best estimates that the probability that Christianity is true were higher than some of the probabilities that show up in the area of x-risk.
I guess I would say that humans are sufficiently bad at reasoning about small probability events when the estimates are not strongly data driven that acting based on such estimates without having somewhat independent arguments for the same action is likely a far mode failure. I'm particularly concerned about the Availability heuristic here.
Sure, most people are not unitary total utilitarians.
I'm unclear on whether my intuition here is coming from a deviation from unitary total utilitarianism or whether my intuition is coming from, e.g. game theoretic considerations that I don't understand explicitly but which are compatible with unitary total utilitarianism.
we have strong negative associations with weak execution, which are pretty well grounded, since one can usually find something trying to do the same task more efficiently.
Agree.
That applies to x-risk as well.
Except insofar as the there's relatively little interest in x-risk and few organizations involved (again, not making a judgment about particular organizations here).
The meaningful question is: "considering the best way to reduce existential risk I can find, including investing in the creation or identification of new opportunities and holding resources in hope of finding such in the future, do I prefer it to some charity that reduces existential risk less but displays more indicators of virtue and benefits current people in the near-term in conventional ways more?"
My own intuition points me toward favoring a focus on existential risk reduction but I have uncertainty as to whether it's right (at least for me personally) because:
(i) I've found thinking about existential risk reduction on account of the poor quality of the information available and the multitude of relevant considerations. As Anna says in Making your explicit reasoning trustworthy:
Some people find their beliefs changing rapidly back and forth, based for example on the particular lines of argument they're currently pondering, or the beliefs of those they've recently read or talked to. Such fluctuations are generally bad news for both the accuracy of your beliefs and the usefulness of your actions.
(ii) Most of the people who I know are not in favor of near-term overt focus on existential risk reduction. I don't know whether this is because I have implicit knowledge that they don't have, because they have implicit knowledge that I don't have or because they're motivated to be opposed to such near-term overt focus for reasons unrelated to global welfare. I lean toward thinking that the situation is some combination of the latter two of the three. I'm quite confused about this matter.
Most of the people who I know are not in favor of near-term overt focus on existential risk reduction. I don't know whether this is because I have implicit knowledge that they don't have, because they have implicit knowledge that I don't have or because they're motivated to be opposed to such near-term overt focus for reasons unrelated to global welfare. I lean toward thinking that the situation is some combination of the latter two of the three.
I think you would normally expect genuine concern about saving the world to be rare among evolved creatures. ...
Over the past six months I've been repeatedly going back and forth on my attitude toward the value of short-term and/or exclusive focus on existential risk. Here I'll offer some reasons why a utilitarian who recognizes the upside of preventing human extinction may refrain from a direct focus on existential risk reduction. I remain undecided on my attitude toward short-term and/or exclusive focus on existential risk - this article is not rhetorical in intent; I'm just throwing some relevant issues out there.
1. On the subject of FAI research, Prase stated that:
The same can be said of much of the speculation concerning existential risk in general, not so much existential risk due to asteroid strike or Venusian global warming but rather with the higher probability but much more amorphous existential risks connected with advanced technologies (general artificial intelligence, whole brain emulation, nano weapons, genetically engineered viruses, etc.).
A principle widely held by many highly educated people is that it's virtually impossible to predict the future more than a few decades out. Now, one can attempt to quantify "virtually impossible" as a small probability that one's model of the future is correct and multiply it by the numbers that emerge as outputs of one's model of the future in Fermi calculations, but the multiplier corresponding to "virtually impossible" may be considerably smaller than one might naively suppose...
2. As AnnaSalamon said in Goals for which Less Wrong does (and doesn't) help,
Assuming that A and B are independent events, the probability of their conjunction is p(A)p(B). So for example, an event that's the conjunction of n independent events each with probability 0.1 occurs with probability 10-n. As humans are systematically biased toward believing that conjunctions are more likely than their conjuncts (at least in certain setting), there's a strong possibility of exponentially overestimating probabilities in the course of Fermi calculations. This is true both of the probability that one's model is correct (given the amount of uncertainty involved in the future as reflected by historical precedent) and of the individual probabilities involved assuming that one's model is correct.
Note that I'm not casting doubt on the utility of Fermi calculations as a general matter - Carl Shulman has been writing an interesting series of posts arguing that one can use Fermi calculations to draw reasonable conclusions about political advocacy as philanthropy. However, Carl's posts have been data-driven in a much stronger sense than Fermi calculations about the probabilities of technologically driven existential risks have been.
3. While the efficient market hypothesis may not hold in the context of philanthropy, it's arguable that the philanthropic world is efficient given the human resources and social institutions that are on the table. Majoritarianism is epistemically wrong, but society is quite rigid and whether or not successful advocacy of a particular cause is tenable depends in some measure on whether society is ready for it. In Public Choice and the Altruist's Burden Roko wrote
Even when epistemically justified in the abstract, focus on fringe causes may take too much of a psychological toll on serious supporters in order for serious supporters to be effective in pursuing their goals. To the extent that focus on existential risk requires radical self sacrificing altruism there are dangers of the type described in a comment by Carl Shulman:
4. Because of the upside of ensuring the survival rate is so huge, there's an implicit world view among certain people on Less Wrong that, e.g. existential risk reduction charities offer the opportunities for optimal philanthropy. I think that existential risk reduction charities may offer opportunities for optimal philanthropy, but that the premise that this is so largely independently of the quality of the work that these charities are doing is essentially parallel to the premise behind Pascal's Wager. In Making your explicit reasoning trustworthy Anna Salamon wrote
I'm not able to offer a strong logical argument against the use of Pascal's wager or infinite ethics but nevertheless feel right to reject them as absurd. Similarly, though I'm unable to offer a strong logical argument for doing so (although I've listed some of the relevant intuitions above), I feel right to restrict support to existential risk reduction opportunities that meet some minimal standard for "sufficiently well-conceived and compelling" well above that of multiplying the value of ensuring human survival by a crude guess as to the probability that a given intervention will succeed.
Intuitively, the position "it doesn't matter how well executed charity X's activities are; since charity X is an existential risk reduction charity, charity X triumphs non-existential risk charities" is for me a reductio ad absurdem for adopting a conscious, explicit, single-minded focus on existential risk reduction.
Disclaimer: I do not intend for my comments about the necessity of meeting a minimal standard to apply specifically to any existential risk reduction charity on the table. I have huge uncertainties as to the significance of most of the points that I make in this post. Depending on one's assessment of their significance one could end up either in favor or against short-term and/or explicit focus on existential risk reduction