= 27d567bc2d48d9f40e9ccc20ea370b15)
see comments on Is GiveWell.org the best charity (excluding SIAI)? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (59)
I respect your intention, and I don't want to hijack your thread. But if you feel like explaining, I'm interested in the details. Are your concerns about SIAI in particular, or all existential risks charities? For example, the Future of Humanity Institute at Oxford University accepts donations, and uses those donations, in part, to fund further research into the causes of existential risk, and avenues for reducing existential risk. There are also groups aimed at reducing the risk of global nuclear war, and the risk of biotech disasters.
A second question is how to allocate your research efforts. As you note, the huge differences between different charities’ efficiency mean that research into where to donate can have huge value (relative to e.g. spending more hours earning money). Do you plan to do further research as time passes? If so, what endeavors are on your list of “types of philanthropy worth looking into”?
A major problem of many future existential threat charities is evaluating whether they actually are reducing existential risk, or whether they will actually increase it. The evidence of history, for example, indicates that even the best foreign policy experts are not very good at evaluating a policy's secondary effects and perverse incentives. The result that it is very hard to evaluate whether the net effect of spending money on what is supposed to be "reducing the risk of global thermonuclear war" will actually increase or decrease the risk of global thermonuclear war. The very same multipliers that leverage to massive utility on the assumption of intended consequences leverage to massive disutility on the assumption of perverse consequences.
On the other hand, it's rather easy to evaluate the net value of a heavily-studied vaccination against an endemic disease, and you can be reasonably certain you're not actually spreading the very disease you're trying to fight.
This sounds more like a bias (people want to know they're successful; ambiguity aversion; want a solid warm glow) than a consideration which could reasonably compete against the economies of scale in existential risk. The penalty for increased uncertainty owing to difficulty of measurement is simply not as large as the orders of magnitude of scope involved.
The orders of magnitude of scope can only matter if you know which way they fall. If a donation to the Nuclear Threat initiative increases the risk of global nuclear war (like a reduction in arsenals deceiving a leader into believing he can make a successful first strike), the orders of magnitude of negative result make it a vastly worse choice than burning a hundred-dollar bill just to see the pretty colors.
If something has a 1% chance of working and a .8% chance of backfiring, the expected utility is the same as what it would be if there was a .2% chance of working, assuming that the benefits and harms are equal and opposite. this has only reduced the expected utility be a factor of 5. Existential risk is so important that it is better than many other things by much more than a factor of 5.
Sure. Now, show me the detailed analysis as to how you got those very precise numbers of your specific proposed intervention in existential risk having a 1.00% chance of working and 0.80% chance of backfiring, instead of the opposite numbers of 0.80% working and 1.00% backfiring.
Because, see, if the odds are the second way, then the expected utility of your intervention is massively, massively negative. Existential risk is so important that while reducing it is better than many other things by much more than a factor of five, increasing it is much, much worse than many evils by much more than a factor of five.
The real universe has no "good intentions" exception, but there's a cognitive bias which causes people to overestimate the likelihood that an act taken with good intentions will produce the intended good result and underestimate the risks of negative results. When uncorrected for in matters of existential risk, the result could be, because of the mathematics of scope, an unintentional atrocity.
Now, my back-of-the-envelope calculation is that SIAI doesn't actually increase the risk of an unfriendly AI by actively trying to create friendly AI. There are so many people doing AI anyway, and the default result is so likely to be unfriendly, that SIAI is a decent choice of existential risk charity. If it succeeds, we have upside; if it creates an unfriendly AI, we were screwed anyway.
On the other hand, the Nuclear Threat Initiative is not merely fucking around with what has seemingly been shown to be a fairly stable system in a quest to achieve an endpoint that is itself unlikely to be stable (total nuclear disarmament; the official goal of the NPT), with all sorts of very-hard-to-calculate scenarios which mean it might on net increase risks of nuclear annihilation of humanity. No, it also might be increasing existential threat of, say, runaway greenhouse warming, by secondary discouraging effects on (for example) nuclear power production. There is nobody on the planet who understands human society and economics and power production and everything else involved well enough to say with any confidence whatsoever that a donation to the Nuclear Threat Initiative will have positive expected utility. All we have to go on is the good intentions of the NTI people, which is no more a guarantee than assurances from a local newspaper horoscope that "Your efforts will be rewarded."
It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more about the SIAI might have an even higher expected utility. However, once you have all the relevant information you can get, it is nonsensical say that you can't calculate the true probability; probability is a quantification of your own knowledge.
Even if you believe that someone else knows something that you don't, you must make a best guess (well, a best probability distribution of guesses) and make a judgment under uncertainty. People have brought up the possibilities of information being intentionally withheld and of wishful thinking. These are no excuse; account for them to the best of your ability and choose. No matter how unfair the set of evidence you receive is, there is always an optimal probability distribution over what it really means. This is what is used in expected utility calculations.
Yes, that's what you do. And my analysis is that the best decision under the available uncertainty is that the probability of donating to NTI doing massive good is not distinguishable from the probability of it doing massive harm. The case for 1.0 vs. 0.8 is not any more convincing to me than the case for 0.8 vs. 1.0. Given a hundred questions on the level of whether the Nuclear Threat Initiative is a good thing to do or not, I would not expect my answers to have any more chance of being right than if I answered based entirely on the results of a fair coin. I would, as I said elsewhere in this discussion, take an even-money bet on either side of reality, in the fullness of time, proving the result either is massive weal or massive woe. The massiveness on either side is meaningless because both sides cancel out. The expected utility of a donation to the NTI is, by my estimates, accordingly zero.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size. Anyone opining on whether the NTI is a good or bad idea is, in my opinion, just as foolish as Ptolemy opining on whether the Indian Ocean was enclosed by land in the south. I don't know, you don't know, nobody on Earth knows enough to privilege any hypothesis about the value of NTI above any other.
When you don't know enough to privilege any particular hypothesis over any other, the sheer scale of the possible results doesn't magically create a reason to act.
Your conclusion follows from your premises.
I find some of the description of your state of knowledge doubtful.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
We run into the Gambler's Ruin pretty quickly when dealing with bets concerning existential risk reduction, so the assumption that the benefits and harms are equal and opposite seems questionable. Expected utility calculations need a lot of tweaks in cases like this.
I was not suggesting that this is the actual math; I was merely giving an example to show that the possibility of an existential risk reduction effort backfiring does not necessarily make it a bad idea to contribute.
Say I assign a 0.2% probability to a given intervention averting human extinction. If I assign it a 0.1% probability of bringing about extinction (which otherwise would not have occurred), then I've lost half the value of an intervention with a 0.2% probability of success and no risk of backfire. A 0.198% probability of extinction would leave a hundredth of the value.
Even at that point, it seems like quite a stretch to say that the best estimate of the Nuclear Threat Initiative's existential risk impact is that it is 99% as likely to bring about existential catastrophe as to prevent it. And note that if the risk of backfire is to wipe out more orders of magnitude of x-risk reduction opportunity, the positives and negatives need to be very finely balanced:
Also, as Anna notes, uncertainty on the numbers at this scale leads to high value of information.
If you assigned a 0.2% probability to a social intervention producing a specific result, I'd mostly be highly skeptical that you have enough, good enough, data to put that precise a number on it. Once probabilities get small enough, they're too small for the human brain to accurately estimate them.
To be neutral in reality, yes, the probability must be in a very narrow range. To be neutral within the ability of a human brain to evaluate without systematic quantitative study, it just needs to be small enough that you can't really tell if you're in case 1 or case 2.
Do you mean that people tend to be poorly calibrated? You might mean that events or statements to which people assign 0.2% probability happen more often than that. Or you might mean that they happen less often. But either way one should then shift one's probability estimates to take that information into account.
Or do you mean that such a number would be unstable in response to new information, more thinking about it, getting info on how priming and biases affect estimation (obviously such estimates on the spot depend on noisy factors, and studies show gains just from averaging estimates one makes at different times and so forth), etc?
In either case, if you were compelled to offer betting odds on thousands of independent claims like that (without knowledge of which side of the bet you'd have to take, and otherwise structured to make giving your best estimate the winning strategy) how would you do it?
As an aside, Yvain's post on probability estimation seems relevant here.
The second.
Specifically, I am of the opinion that it is well-demonstrated that calculating adverse consequences of social policy is both sufficiently complicated and sufficiently subject to priming and biases that it is beyond human capacity at this time to accurately estimate whether the well-intentioned efforts of the Nuclear Threat Initiative are more likely to reduce or increase the risk of global thermonuclear war.
If I were forced to take a bet on the issue, I would set the odds at perfectly even. Not because I expect that a full and complete analysis by, say, Omega would come up with the probability being even, but because I have no ability to predict whether Omega would find that the Nuclear Threat initiative reduces or increases the chance of global thermonuclear war.