= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on Earning to Give vs. Altruistic Career Choice Revisited - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (154)
I can imagine someone thinking that FHI was a better use of money than MIRI, or CFAR, or CSER, or the Foresight Institute, or brain-scanning neuroscience, or rapid-response vaccines, or any number of startups, but considering AMF as being in the running at all seems to require either a value difference or really really different epistemics about what affects the fate of future galaxies.
Realistic amounts of difference in epistemics + the "humans best stick to the mainline probability" heuristic seem enough (where by "realistic" I mean "of the degree actually found in the world"). I.e., I honestly believe that there are many people out there who would care the hell about the fate of future galaxies if they alieved that they had any non-vanishing chance of significantly influencing that fate (and to choose the intervention that influences it in the desired direction).
If you're one of 10^11 sentients to be born on Ancient Earth with a golden opportunity to influence a roughly 10^80-sized future, what exactly is a 'vanishing chance'... eh, let's all save it until later.
I meant that the alieved probability is small in absolute terms, not that it is small compared to the payoff. That's why I mentioned the "stick to the mainline probability" heuristic. I really do believe that there are many people who, if they alieved that they (or a group effort they could join) could change the probability of a 10^80-sized future by 10%, would really care; but who do not alieve that the probability is large enough to even register, as a probability; and whose brains will not attempt to multiply a not-even-registering probability with a humongous payoff. (By "alieving a probability" I simply mean processing the scenario the way one's brain processes things it assigns that amount of credence, not a conscious statement about percentages.)
This is meant as a statement about people's actual reasoning processes, not about what would be reasonable (though I did think that you didn't feel that multiplying a very small success probability with a very large payoff was a good reason to donate to MIRI; in any case seems to me that the more important unreasonableness is requesting mountains of evidence before alieving a non-vanishing probability for weird-sounding things).
[ETA: I find it hard to put a number on the not-even-registering probability the sort of person I have in mind might actually alieve, but I think a fair comparison is, say, the "LHC will create black holes" thing -- I think people will tend to process both in a similar way, and this does not mean that they would shrug it off if somebody counterfactually actually did drop a mountain of evidence about either possibility on their head.]
Because on a planet like this one, there ought to be some medium-probable way for you and a cohort of like-minded people to do something about x-risk, and if a particular path seems low probability, you should look for one that's at least medium-probability instead.
Ok, fair enough. (I had misunderstood you on that particular point, sorry.)
If there was ever a reliable indicator that you're wrong about something, it is the belief that you are special to the order of 1 in 10^70.
So do you believe in the Simulation Hypothesis or the Doomsday Argument, then? All attempts to cash out that refusal-to-believe end in one or the other, inevitably.
From where I stand, it's more like arcane meta-arguments about probability are motivating a refusal-to-doubt the assumptions of a prized scenario.
Yes, I am apriori skeptical of anything which says I am that special. I know there are weird counterarguments (SIA) and I never got to the bottom of that debate. But meta issues aside, why should the "10^80 scenario" be the rational default estimation of Earth's significance in the universe?
The 10^80 scenario assumes that it's physically possible to conquer the universe and that nothing would try to stop such a conquest, both enormous assumptions... astronomically naive and optimistic, about the cosmic prospects that await an Earth which doesn't destroy itself.
Okay, so that's the Doomsday Argument then: Since being able to conquer the universe implies we're 10^70 special, we must not be able to conquer the universe.
Calling the converse of this an arcane meta-argument about probability hardly seems fair. You can make a case for Doomsday but it's not non-arcane.
Perhaps this is hairsplitting but the principle I am employing is not arcane: it is that I should doubt theories which imply astronomically improbable things. The only unusual step is to realize that theories with vast future populations have such an implication.
I am unable to state what the SIA counterargument is.
Right. That's arcane. Mundane theories have no need to measure the population of the universe.
But it's still a simple idea once you grasp it. I was hoping you could state the counterargument with comparable simplicity. What is the counterargument at the level of principles, which neutralizes this one?
In the theory that there are astronomically large numbers of people, it is a certainty that some of them came first. The probability that YOU are one of those people equal to the probability that YOU are any one of those other people. However, it does define a certain small narrow equivalence class that you happen to be a member of.
It's a bit like the difference between theorizing that: A) given that you bought a ticket, you'll win the lottery, and B) given that the lottery folks gave you a large sum, that you had the winning ticket.
That's not the "SIA counterargument", which is what I want to hear (in a compact form, that makes it sound straightforward). You're just saying "accept the evidence that something ultra-improbable happened to you, because it had to happen to someone".
I largely agree with your skepticism. I would go even farther and say that even the 10^80 scenario happens, what we do now can only influence it by random chance, because the uncertainty in the calculations of the consequences of our actions in the near term on the far future overwhelms the calculations themselves. That said, we should still do what we think is best in the near term (defined by our estimates of the uncertainty being reasonably small), just not invoke the 10^80 leverage argument. This can probably be formalized, by assuming that the prediction error grows exponentially with some relevant parameter, like time or the number of choices investigated, and calculating the exponent from historical data.
Doomsday for me, I think. Especially when you consider that it doesn't mean doomsday is literally imminent, just "imminent" relative to the kind of timescale that would be expected to create populations on the order of 10^80.
In other words, it fits with the default human assumption that civilization will basically continue as it is for another few centuries or millennia before being wiped out by some great catastrophe.
Do you mind elaborating on this inevitability? It seems like there ought to be other assumptions involved. For example, I can easily imagine that humans will never be able to colonize even this one galaxy, or even any solar system other than this one. Or that they will artificially limit the number of individuals. Or maybe the only consistent CEV is that of a single superintelligence of which human minds will be tiny parts. All of these result in the rather small total number of individuals existing at any point in time.
Counts as Doomsday, also doesn't work because this solar system could support vast numbers of uploads for vast amounts of time (by comparison to previous population).
This is a potential reply to both Doomsday and SA but only if you think that 'random individual' has more force than a similar argument from 'random observer-moment', i.e. to the second you reply, "What do you mean, why am I near the beginning of a billion-year life rather than the middle? Anyone would think that near the beginning!" (And then you have to not translate that argument back into a beginning-civilization saying the same thing.)
...whereupon we wonder something about total 'experience mass', and, if that argument doesn't go through, why the original Doomsday Argument / SH should either.
Thanks, I'll chew on that a bit. I don't understand the argument in the second and third paragraphs. Also, it's not clear to me whether by "counts as doomsday" you mean the standard doomsday with the probability estimates attached, or some generalized doomsday, with no clear timeline or total number of people estimated.
Anyway, the feeling I get from your reply is that I'm missing some basic background stuff here I need to go through first, not the usual "this guy is talking out of his ass" impression when someone invokes anthropics in an argument.
No, this is talking-out-of-our-ass anthropics, it's just that the anthropic part comes in when you start arguing "No, you can't really be in a position of that much influence", not when you're shrugging "Sure, why shouldn't you have that much influence?" Like, if you're not arriving at your probability estimate for "Humans will never leave the solar system" just by looking at the costs of interstellar travel, and are factoring in how unique we'd have to be, this is where the talking-out-of-our-ass anthropics comes in.
Though it should be clearly stated that, as always, "We don't need to talk out of our ass!" is also talking out of your ass, and not necessarily a nicer ass.
Or when you (the generic you) start arguing "Yes, I am indeed in a position of that much influence", as opposed to "There is an unknown chance of me being in such a position, which I cannot give a ballpark estimate for without talking out of my ass, so I won't"?
I've had a vague idea as to why the random observer-moment argument might not be as strong as the random individual one, though I'm not very confident it makes much sense. (But neither argument sounds anywhere near obviously wrong to me.)
I wonder if this argument can be made precise enough to have its premises and all the intermediate assumptions examined. I remain skeptical of any forecast that far into the future. You presumably mean your confidence in the UFAI x-risk within the next 20-100 years as the minimum hurdle to overcome, with the eternal FAI paradise to follow.
My reason for mentioning AMF and global health is that doing so provides a concrete, pretty robustly researched example, rather than as to compare with efforts to improve the far future of humanity.
I think that working in global health in a reflective and goal directed way is probably better for improving global health than "earning to give" to AMF. Similarly, I think that working directly on things that bear on the long term future of humanity is probably a better way of improving the far future of humanity than "earning to give" to efforts along these lines.
I'll discuss particular opportunities to impact the far future of humanity later on.
That depends on what you want to know, doesn't it? As far as I know the impact of AMF on x-risk, astronomical waste, and total utilons integrated over the future of the galaxies, is very poorly researched and not at all concrete. Perhaps some other fact about AMF is concrete and robustly researched, but is it the fact I need for my decision-making?
(Yes, let's talk about this later on. I'm sorry to be bothersome but talking about AMF in the same breath as x-risk just seems really odd. The key issues are going to be very different when you're trying to do something so near-term, established, without scary ambiguity, etc. as AMF.)
I'm somewhat confused by the direction that this discussion has taken. I might be missing something, but I believe that the points related to AMF that I've made are:
GiveWell's explicit cost-effectiveness estimate for AMF is much higher than the cost per DALY saved implied by the figure that MacAskill cited.
GiveWell's explicit estimates for the cost-effectiveness of the best giving opportunities in the field of direct global health interventions have steadily gotten lower, and by conservation of expected evidence, one can expect this trend to continue.
The degree of regression to the mean observed in practice suggests that there's less variance amongst the cost-effectiveness of giving opportunities than may initially appear to be the case.
By choosing an altruistic career path, one can cut down on the number of small probability failure modes associated with what you do.
I don't remember mentioning AMF and x-risk reduction together at all. I recognize that it's in principle possible that the "earning to give" route is better for x-risk reduction than it is for improving global health, but I believe the analogy between the two domains is sufficiently strong that my remarks on AMF have relevance (on a meta-level, not on an object level).
Yeah, I also have the feeling that I'm questioning you improperly in some fashion. I'm mostly driven by a sense that AMF is very disanalogous to the choices that face somebody trying to optimize x-risk charity (or rather total utilons over all future time, but x-risk seems to be the word we use for that nowadays). It seems though that we're trying to have a discussion in an ad-hoc fashion that should be tabled and delayed for explicit discussion in a future post, as you say.
If I may list some differences I perceive between AMF and MIRI:
Near mode thinking will most likely direct one to AMF. MIRI probably requires one to shut up and multiply. Which is probably why I'm currently giving a little money to Greenpeace, despite being increasingly certain that it's far, far from the best choice.
One more difference:
AMF's impact is very likely to be net positive for the world under all reasonable hypotheses.
MIRI appears to me to have a chance to be massively net negative for humanity. I.e. if AI of the level they predict is actually possible, MIRI might end up creating or assisting in the creation of UFAI that would not otherwise be created, or perhaps not created as soon.
But what if AMF saves a child who grows up to be a biotechnologist and goes on to weaponize malaria and spread it to millions?
If you try hard enough, you can tell a story where any effort to accomplish X somehow turns out to accomplish ~X, but one must distinguish possibility from the balance of probability.
That seems like a bizarre belief to hold. Or perhaps just overwhelmingly shortsighted. There are certainly reasonable hypotheses in which more people alive right now result in worse outcomes a single generation down the line, without even considering extinction level threats and opportunities. The world isn't nearly easy enough to model and optimize for us to be that certain a disruptive influence on that scale will be a net positive under all reasonable hypotheses.
Would you care to cite any such reasonable hypotheses? I.e. under what assumptions do you think that saving a random poor person's life is likely to be a net negative? Sum over the number of lives saved and even if one person grows up to be a serial killer, the total is still way positive. Can you really defend a situation in which it is preferable to have living people today die from malaria?
The problem with MIRI-hypothesized AI (beyond its implausibility) is that we don't get to sum over all possible results. We get one result. Even if the chance of a good result is 80%, the chance of a disastrous result is still way too high for comfort.
Most obviously it could cause an increase in world GDP without a commensurate acceleration in various risk prevention mechanisms. Species can evolve themselves to extinction and in a similar way humans could easily develop themselves to extinction if they are not careful or lucky. Messing around with various aspects of the human population would influence this... in one direction or another. It's damn hard to predict.
Having a heuristic "short term lives saved == good" is useful. It massively simplifies calculations and if you have no information either way about side effects of the influence then it works well enough. But it would a significant epistemic error to mistake the heuristic for operating under uncertainty with confidence about the unpredictable (or difficult to predict) system in which you are operating.
What is socially defensible is not the same thing as what is accurate. But that isn't the point here. All else being equal I would prefer AMF to have an extra million dollars to spend than to not have that extra million dollars. The expected value is positive. What I criticise is "very likely under all reasonable hypotheses" which is just way off. I do not have the epistemic resources to arrive at that confidence and I believe that you are arriving at that conclusion in error, not because of additional knowledge or probabilistic computational resources.
In fact, I'd expect AMF to have a net-negative impact (and a large one at that) a few decades down the line, unless there are unrealistic, unprecedented, imperialistic-in-scope, gigantic efforts to educate and provide for the dozen then-adult children (and their dozen children) a saved-from-malaria child can typically have.
Here's Tom Friedman in his recent "Tell Me How This Ends" column:
Do you really want to propose that it is better to let children in poor countries die of disease now than to save them, because they might have more children later? My prior on this is that you're trolling, but if you really believe that and are willing to state it that baldly; then it might be worth having a serious conversation about population.
I'll grant that MIRI could accelerate the creation of AGI, if their efforts to educate people about UFAI risks are particularly ineffective. But as far as UFAI creation at all is concerned, there are any number of very smart idiots in the world who would love to be on the news as "the first person to program an artificial general intelligence". Or to be the first person to use a general AI to beat the stock market, as soon as enough parts of the puzzle have been worked out to make one by pasting together published math results. (Maybe a slightly more self-aware variation of AIXI-mc would do the trick.)
In my view, AGI is more or less inevitable, and MIRI is seemingly the only group publically interested in making it safe.
Not really related to the current discussion, but I want to make sure I understand the above statement. Is this assuming that the trend has not already been taken into account in forming the estimates?
Yes — the cost-effectiveness estimate has been adjusted every time a new issue has arisen, but on a case by case basis, without an attempt to extrapolate based on the historical trend.