= 27d567bc2d48d9f40e9ccc20ea370b15)
peter_hurford comments on The $125,000 Summer Singularity Challenge - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (259)
I understand the SI needs money and I understand a lot of discussion about this has ensued elsewhere, but I'm still skeptical that I can have the most impact with my money by donating to the SI, when I could be funding malaria nets, for instance.
There are two questions here that deserve separate consideration: donating to existential risk reduction vs. other (nearer-term, lower-uncertainty) philanthropy, and donating to SI vs. other x-risk reduction efforts. It seems to me that you should never be weighing SI against malaria nets directly; if you would donate to (SI / malaria nets) conditional on their effectiveness, you've already decided (for / against) x-risk reduction and should only be considering alternatives like (FHI / vaccination programs).
Thanks. You're right I've been thinking about it wrong, I'll have to reconsider how I approach philanthropy. It's valuable to donate to research anyway, since research is what comes up with things like "malaria nets".
Glad I could help. Thanks for letting me know.
Good point; under uncertainty about x-risk vs. near-term philanthropy you might donate to organizations that could help answer that question, like GiveWell or SI/FHI.
They planned on doing an academic paper on the topic, though it hasn't been completed yet. Here's Anna Salamon's presentation, estimating 8 lives saved per dollar donated to SingInst.
8 lives per dollar is an awful, awful lot, but I'll definitely check out those resources. If the 8 lives per dollar claim is true, I'll be spending my money on SI.
If a back-of-the-envelope calculation comes up with a number like that, then it is probably wrong.
I haven't watched the presentation, but 8 lives corresponds to only a one in a billion chance of averting human extinction per donated dollar, which corresponds (neglecting donation matching and the diminishing marginal value of money) to roughly a 1 in 2000 chance of averting human extinction from a doubling of the organization's budget for a year. That doesn't sound obviously crazy to me, though it's more than I'd attribute to an organization just on the basis that it claimed to be reducing extinction risk.
For what it's worth, this is in line with my estimates, which are not just on the basis of claimed interest in x-risk reduction. I don't think that an order of magnitude or more less than this level of effectiveness could be the conclusion of a credible estimation procedure.
The topics of existential risk, AI, and other future technologies inherently require the use of very large numbers, far beyond any of those encountered when discussing normal, everyday risks and rewards.
Note that the large number used in this particular back-of-envelope calculation is the world population of several billion, not the still much larger numbers involved in astronomical waste.
Even if this is so, there is tons of evidence that humans suck at reasoning about such large numbers. If you want to make an extraordinary claim like the one you made above, then you need to put forth a large amount of evidence to support it. And on such a far-mode topic, the likelihood of your argument being correct decreases exponentially with the number of steps in the inferential chain.
I only skimmed through the video, but assuming that the estimates at 11:36 are what you're referring to, those numbers are both seemingly quite high and entirely unjustified in the presentation. It also overlooks things like the fact that utility doesn't scale linearly in number of lives saved when calculating the benefit per dollar.
Whether or not those numbers are correct, presenting them in their current form seems unlikely to be very productive. Likely either the person you are talking to already agrees, or the 8 lives figure triggers an absurdity heuristic that will demand large amounts of evidence. Heck, I'm already pretty familiar with the arguments, and I still get a small amount of negative affect whenever someone tries to make the "donating to X-risk has <insert very large number> expected utility".
I don't think anyone on LW disagrees that reducing xrisk substantially carries an extremely high utility. The points of disagreement are over whether SIAI can non-trivially reduce xrisk, and whether they are the most effective way to do so. At least on this website, this seems like the more productive path of discussion.
Keep in mind that estimation is the best we have. You can't appeal to Nature for not having been given a warning that meets a sufficient standard of rigor. Avoiding all actions of uncertain character dealing with huge consequences is certainly a bad strategy. Any one of such actions might have a big chance of not working out, but not taking any of them is guaranteed to be unhelpful.
From a Bayesian point of view, your prior should place low probability on a figure like "8 lives per dollar". Therefore, lots of evidence is required to overcome that prior.
From a decision-theoretic point of view, the general strategy of believing sketchy (with no offense intended to Anna; I look forward to reading the paper when it is written) arguments that reach extreme conclusions at the end is a bad strategy. There would have to be a reason why this argument was somehow different from all other arguments of this form.
If there were tons of actions lying around with similarly huge potential positive consequences, then I would be first in line to take them (for exactly the reason you gave). As it stands, it seems like in reality I get a one-time chance to reduce p(bad singularity) by some small amount. More explicitly, it seems like SIAI's research program reduces xrisk by some small amount, and a handful of other programs would also reduce xrisk by some small amount. There is no combined set of programs that cumulatively reduces xrisk by some large amount (say > 3% to be explicit).
I have to admit that I'm a little bit confused about how to reason here. The issue is that any action I can personally take will only decrease xrisk by some small amount anyways. But to me the situation feels different if society can collectively decrease xrisk by some large amount, versus if even collectively we can only decrease it by some small amount. My current estimate is that we are in the latter case, not the former --- even if xrisk research had unlimited funding, we could only decrease total xrisk by something like 1%. My intuitions here are further complicated by the fact that I also think humans are very bad at estimating small probabilities --- so the 1% figure could very easily be a gross overestimate, whereas I think a 5% figure is starting to get into the range where humans are a bit better at estimating, and is less likely to be such a bad overestimate.
My prior contains no such provisions; there are many possible worlds where tiny applications of resources have apparently disproportionate effect, and from the outside they don't look so unlikely to me.
There are good reasons to be suspicious of claims of unusual effectiveness, but I recommend making that reasoning explicit and seeing what it says about this situation and how strongly.
There are also good reasons to be suspicious of arguments involving tiny probabilities, but keep in mind: first, you probably aren't 97% confident that we have so little control over the future (I've thought about it a lot and am much more optimistic), and second, that even in a pessimistic scenario it is clearly worth thinking seriously about how to handle this sort of uncertainty, because there is quite a lot to gain.
Of course this isn't an argument that you should support the SIAI in particular (though it may be worth doing some information-gathering to understand what they are currently doing), but that you should continue to optimize in good faith.
Can you clarify what you mean by this?
Only that you consider the arguments you have advanced in good faith, as a difficulty and a piece of evidence rather than potential excuses.
I'm glad you agree.
I'd be very appreciative to hear if you know of someone doing more.
Over the coming months I'm going to be doing an investigation of the non-profits affiliated with the Nuclear Threat Initiative with a view toward finding x-risk reduction charities other than SIAI & FHI. I'll report back what I learn but it may be a while.
I'm under the impression that nuclear war doesn't pose an existential risk. Do you disagree? If so, I probably ought to make a discussion post on the subject so we don't take this one too far off topic.
My impression is that the risk of immediate extinction due to nuclear war is very small but that a nuclear war could cripple civilization to the point of not being able to recover enough to affect a positive singularity; also it would plausibly increase other x-risks - intuitively, nuclear war would destabilize society, and people are less likely to take safety precautions in an unstable society when developing advanced technologies than they otherwise would be. I'd give a subjective estimate of 0.1% - 1% of nuclear war preventing a positive singularity.
Do you mean:
Thanks for the clarification on the estimate. Unhappy as it makes me to say it, I suspect that nuclear war or other non-existential catastrophe would overall reduce existential risk, because we'd have more time to think about existential risk mitigation while we rebuild society. However I suspect that trying to bring nuclear war about as a result of this reasoning is not a winning strategy.
Well for instance, certain approaches to AGI are more likely to lead to something friendly than other approaches are. If you believe that approach A is 1% less likely to lead to a bad outcome than approach B, then funding research in approach A is already compelling.
In my mind, a well-reasoned statistical approach with good software engineering methodologies is the mainstream approach that is least likely to lead to a bad outcome. It has the advantage that there is already a large amount of related research being done, hence there is actually a reasonable chance that such an AGI would be the first to be implemented. My personal estimate is that such an approach carries about 10% less risk than an alternative approach where the statistics and software are both hacked together.
In contrast, I estimate that SIAI's FAI approach would carry about 90% less risk if implemented than a hacked-together AGI. However, I assign very low probability to SIAI's current approach succeeding in time. I therefore consider the above-mentioned approach more effective.
Another alternative to SIAI that doesn't require estimates about any specific research program would be to fund the creation of high-status AI researchers who care about Friendliness. Then they are free to steer the field as a whole towards whatever direction is determined to carry the least risk, after we have the chance to do further research to determine that direction.
I don't understand what you mean by "10% less risk". Do you think any given project using "a well-reasoned statistical approach with good software engineering methodologies" has at least 10% chance of leading to a positive Singularity? Or each such project has a P*0.9 probability of causing an existential disaster, where P is the probability of disaster of a "hacked together" project. Or something else?
Sorry for the ambiguity. I meant P*0.9.
You said "I therefore consider the above-mentioned approach more effective.", but if all you're claiming is that the above mentioned approach ("a well-reasoned statistical approach with good software engineering methodologies") has a P*0.9 probability of causing an existential disaster, and not claiming that it has a significant chance of causing a positive Singularity, then why do you think funding such projects is effective for reducing existential risk? Is the idea that each such project would displace a "hacked together" project that would otherwise be started?
I noticed you didn't name anybody. Did you have specific programs or people in mind?
We already seem to (roughly) agree on probabilities.
The only specific plan I have right now is to put myself in a position to hire smart people to work on this problem. I think the most robust way to do this is to get a faculty position somewhere, but I need to consider the higher relative efficiency of corporations over universities some more to figure out if it's worthwhile to go with the higher-volatility route of industry.
Also, as Paul notes, I need to consider other approaches to x-risk reduction as well to see if I can do better than my current plan. The main argument in favor of my current plan is that there is a clear path to the goal, with only modest technical hurdles and no major social hurdles. I don't particularly like plans that start to get fuzzier than that, but I am willing to be convinced that this is irrational.
EDIT: To be more explicit, my current goal is to become one of said high-status AI researchers. I am worried that this is slightly self-serving, although I think I have good reason to believe that I have a comparative advantage at this task.
That seems more of an alternative within SIAI than an alternative to SIAI. With more funding, their Associate Research Program can promote the importance of Friendliness and increase the status of researchers who care about it.
Woah, woah! What! Since when?
Unless you mean "scope insensitivity"?
Well, sure, the absurdity heuristic is terrible.
Why would it scale linearly? I agree that is scales linearly over relatively small regimes (on the order of millions of lives) by fungibility, but I see no reason why that needs to be true for trillions of lives or more (and at least some reasons why it can't scale linearly forever).
Re-read the context of what I wrote. Whether or not the absurdity heuristic is a good heuristic, it is one that is fairly common among humans, so if your goal is to have a productive conversation with someone who doesn't already agree with you, you shouldn't throw out such an ambitious figure without a solid argument. You can almost certainly make whatever point you want to make with more conservative numbers.
Transcript; the precise wording is "You can divide it up, per half day of time, something like 800 lives. Per $100 of funding, also something like 800 lives." It's at 12:31. The slide up at that moment during the presentation emphasises the point, this wasn't a casual aside.
Anna made a comment about this in January 2014.
And for something in the developing world aid space, Village Reach is generally considered to be the most efficient.
http://www.givewell.org/international/top-charities/villagereach
Yeah, I intend to donate a good portion to Village Reach after I do some more thorough research on charity. I don't have that much of an income yet, anyway.
If you already know your decision the value of the research is nil.
Lol, good point. What I meant to say was "I probably intend to donate a good portion to Village Reach, if I don't encounter anything in my research to change my mind." It's still probably a biased approach, but I can't pretend I don't already have a base point for my donations.
No because then if someone challenges your decision you can give them citations! And then you can carry out the decision without the risk of looking weird!
Citing evidence that didn't influence you before you wrote your bottom line is lying.
Added some exclamation marks to bring out the sarcasm.
So if:
Then I'm lying? I don't think that's quite right.
Nah; if your credence in X went up when you read the new reasons, and more importantly if it would have gone down if the opposite of these reasons were true, it's kosher.
If someone challenges your post and you think "Crap, my case doesn't look impressive enough" and selectively search for citations, you're lying.
A grey area is when you believe X because you heard it somewhere but you don't remember where except that it sounded trustworthy. You can legitimately be pretty confident that X is true and that good sources exist, but you still have to learn a new fact before you can point to them. The reason this isn't an outright lie is that trust chains need occasional snapping. There's an odd and interesting effect - Alice distorts things just a tiny bit when she tells Bob, which basically doesn't affect anything, but Bob doesn't know exactly what the distortions where so the distorsions he adds when he tells Carol can be huge, though his beliefs are basically correct! (A big source is that uncertainty is hard to communicate, so wild guesses often turn into strong claims.)
"Selectively" is the keyword here. Searching for additional arguments for your position is legitimate if you would retract on discovering negative evidence IMO.
Yeah, but that's a weird thing to do. Why not give your current evidence, then do more research and come back to announce the results?
No. It just isn't. But adopting your ontology briefly I will assert that 'lying' is morally virtuous in all sorts of situations.
A worthy endeavour!
Are you being sarcastic here?
No. Information really is useful for influencing others independently of its use for actually making decisions. It is only the decision making component that is useless after you have already made up your mind.
Okay, thanks.
Nah, you can't choose to un-donate. Whereas you can always make up for lost time. So giving is a case where some mild indecision may be worthwhile.
Obviously the current expected value of your action should be the same as what you expect to think in the future. But getting more info can increase the expected value of your information thus:
Let's say you have a charity budget, and two charities, A and B. Since your budget is a small fraction of the budget of each charity, assume your utility is linear in this decision, so you'll give all your money to the better charity. You think there's a 60% chance that charity A produces 1000 utilons from your donation and charity B produces 100, and a 40% chance that A only produces 10 and B still produces 100. The expected utility of giving to A is 60% * 1000 + 40% * 10 = 604. The expected utility of giving to B is 60% * 100 + 40% * 100 = 100, so you are planning to give to A.
But let's say that by doing some small amount of research (assuming it's costless for simplicity), you can expect to become, correctly, nearly certain (one situation has probability of ~1, the other has probability of ~0). Now if you become certain that A produces 1000 utilons (which you expect to happen 60% of the time), your choice is the same. But if you become certain that A produces only 10 utilons, you give to B instead. So your expected utility is now 60% * 1000 + 40% * 100 = 640, a net gain of 36 expected utilons.
You seem to have missed the point.
The point being what?
All information gained after making a decision is irrelevant for the purpose of making said decision. See also: The Bottom Line.
Not if you can remake the decision. I read "I intend [...]" to mean "I expect to make this decision, based on the evidence available now, but will gather more evidence first, which may change my mind."
But people don't do very well at thinking when there's already an expected outcome, so peter_hurford should either give up or work on becoming more curious.