Sure. Now, show me the detailed analysis as to how you got those very precise numbers of your specific proposed intervention in existential risk having a 1.00% chance of working and 0.80% chance of backfiring, instead of the opposite numbers of 0.80% working and 1.00% backfiring.
Because, see, if the odds are the second way, then the expected utility of your intervention is massively, massively negative. Existential risk is so important that while reducing it is better than many other things by much more than a factor of five, increasing it is much, much worse than many evils by much more than a factor of five.
The real universe has no "good intentions" exception, but there's a cognitive bias which causes people to overestimate the likelihood that an act taken with good intentions will produce the intended good result and underestimate the risks of negative results. When uncorrected for in matters of existential risk, the result could be, because of the mathematics of scope, an unintentional atrocity.
Now, my back-of-the-envelope calculation is that SIAI doesn't actually increase the risk of an unfriendly AI by actively trying to create friendly AI. There are so many people doing AI anyway, and the default result is so likely to be unfriendly, that SIAI is a decent choice of existential risk charity. If it succeeds, we have upside; if it creates an unfriendly AI, we were screwed anyway.
On the other hand, the Nuclear Threat Initiative is not merely fucking around with what has seemingly been shown to be a fairly stable system in a quest to achieve an endpoint that is itself unlikely to be stable (total nuclear disarmament; the official goal of the NPT), with all sorts of very-hard-to-calculate scenarios which mean it might on net increase risks of nuclear annihilation of humanity. No, it also might be increasing existential threat of, say, runaway greenhouse warming, by secondary discouraging effects on (for example) nuclear power production. There is nobody on the planet who understands human society and economics and power production and everything else involved well enough to say with any confidence whatsoever that a donation to the Nuclear Threat Initiative will have positive expected utility. All we have to go on is the good intentions of the NTI people, which is no more a guarantee than assurances from a local newspaper horoscope that "Your efforts will be rewarded."
It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more...
Update: I should've said "non-existential risk charity", rather than specifically exclude SIAI. I'm having trouble articulating why I don't want to give to an existential risk charity, so I'm going to think more deeply about it. This post is close to my source of discomfort, which is about the many highly uncertain assumptions necessary to motivate existential risk reduction. However, I couldn't articulate this argument properly before, so it might not be the true source of my discomfort. I'll keep thinking.
I received my first pay-cheque from my first job after getting my degree, so it's time to start tithing. So I've been evalating which charity to donate to. I'd like to support the SIAI but I'm not currently convinced it's the best-value charity in a dollars-per-life sense, once time-value of money discounting is applied. I'd like to discuss the best non-SIAI charity available.
By far the best source of information I've found is www.givewell.org. It was started by two hedge fund managers who were struck by the absence of rational charity evaluations, so decided that this was the most pressing problem they could work on.
Perhaps the clearest, deepest finding from the studies they pull together and discuss is that charity is hard. Spending money doesn't automatically translate to doing good. It's not even enough to have smart people who care and know a lot about the problem think of ideas, and then spend money doing them. There's still a good chance the idea won't work. So we need to be evaluating programs rigorously before we scale them up, and keep evaluating as we scale.
The bad news is that this isn't how charity is usually done. Very few charities make convincing evaluations of their activities public, if they carry them out at all. The good news is that some of the programs that have been evaluated are very, very effective. So choosing a charity rationally is absolutely critical.
Let's say you're interested specifically in HIV/AIDS relief.[1] You could fund a program that mainly distributes Anti-Retroviral Therapy to HIV/AIDS patients, which has been estimated conservatively to cost $1494 per disability adjusted life-year (DALY). Alternatively, you could fund a condom distribution program, which has been estimated conservatively to cost $112 per DALY. Or, you could fund a program to prevent mother-to-child transmission, which has been estimated conservatively to cost $12 per DALY. So even within HIV/AIDS, funding the right program can make your donation two orders of magnitude more effective. By tithing 10% of my income every year for the next thirty years, I could have a bigger impact than a $25 million donation, if the person who placed that donation only did an okay job of choosing a charity.
GiveWell currently gives its top recommendation to VillageReach, a charity that seeks to improve logistics for vaccine delivery to remote communities. The evidence is less cut-and-dried than you'd ideally want, but it's still compelling. They took vaccine rates up to 95%, and had very low stock-out rates for vaccines during the 4 year pilot project in Mozambique. They're estimated to have spent about $200usd per life saved. Even if future projects are two or three times less efficient, you're still saving a life for $600. Think about how little money that is. If you tithe, you can probably expect to save 10 lives a year. That's massive.
Instead of donating directly to VillageReach, I'm going to just donate to GiveWell. They pool the funds they get and distribute them to their top charities, and I trust their analytic, evidence-based, largely utilitarian approach. Mostly, however, I think the work they're doing gathering and distributing information about charities is critically important. If more charities actually competed on evidence of efficacy, the whole endeavour might be a lot different. Does anyone have any better suggestions?
[1] I don't understand why people would want to help sufferers of one disease or condition specifically, instead of picking the lowest-hanging fruit, but apparently they do.