Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Where I've Changed My Mind on My Approach to Speculative Causes

36 Post author: peter_hurford 16 August 2013 07:09AM

Follow up to Why I'm Skeptical About Unproven Causes (And You Should Be Too)

Previously, I wrote "Why I'm Skeptical About Unproven Causes (And You Should Be Too)" and a follow up essay "What Would It Take to Prove a Speculative Cause?".  Both of these sparked a lot of discussion on LessWrong, on the Effective Altruist blog, and my own blog, as well as many hours of in person conversation.

After all this extended conversation with people, I've changed my mind on a few things that I will elaborate here.  I hope in doing so I can (1) clarify my original position and (2) explain where I now stand in light of all the debate so people can engage with my current ideas as opposed to the ideas I no longer hold.  My opinions on things tend to change quickly, so I think updates like this will help.

 

My Argument, As It Currently Stands

If I were to communicate one main point of my essay, based on what I believe now, it would be when you're in a position of high uncertainty, the best response is to use a strategy of exploration rather than a strategy of exploitation.

What I mean by this is that given the high uncertainty of impact we see now, especially with regard to the far future, we're better off trying to find more information about impact and reduce our uncertainty (exploration) rather than pursuing whatever we think is best (exploitation).

The implications of this would mean that:

  • We should develop more of an attitude that our case for impact is neither clear nor proven.
  • We should apply more skepticism to our causes and more self-skepticism to our personal beliefs about impact.
  • We should use the language of "information" and "exploration" more often than the language of "impact" and "exploitation".
  • We should focus more on finding specific and concrete attempts to ensure we're making progress and figure out our impact (whether it be surveys, experiments, soliciting external review from relevant experts, etc.).
  • We should focus more on transparency about what we're doing and thinking and why, when relevant and not exceedingly costly.

 

And to be clear, here are specific statements that address misconceptions about what I have argued:

  • I do think it is wrong to ignore unproven causes completely and stop pursuing them.
  • I don't think we should be donating everything to the Against Malaria Foundation instead of speculative causes.
  • I don't think the Against Malaria Foundation has the highest impact of all current opportunities to donate.
  • I do think we can say useful things about the far future.
  • I don't think the correct way to think about high uncertainty and low evidence is to "suspend judgement".  Rather, I think we should make a judgement that we expect the estimate to be much lower than initially claimed in light of all the things I've said earlier about the history of past cost-effectiveness estimates.

 

And, lastly, if I were to make a second important point it would be it's difficult to find good opportunities to buy information.  It's easy to think that any donation to an organization will generate good information or that we'll automatically make progress just by working.  I think some element of random pursuit is important (see below), but all things considered I think we're doing too much random pursuit right now.

 

Specific Things I Changed My Mind About

Here are the specific places I changed my mind on:

 

I used to think donating to AMF, at least in part, was important for me.  Now I don't.

I underestimated the power of exploring and the existing opportunities, so I think that 100% of my donations should be going to trying to assess impact.  I've been persuaded that there is already quite a lot of money going toward AMF and we might not need more money as quickly as thought, so for the time being it's probably more appropriate to save and then donate to opportunities to buy information as they come up.

 

I now agree that there are relevant economies of scale in pursuing information that I hadn't taken into account.

What I mean by this is it might not be appropriate for individuals to work on purchasing information themselves.  Instead, this could end up splitting up the time of organizations unnecessarily as they provide information to a bunch of different people.  Also, many people don't have the time to do this themselves.

I think this has two implications:

  • We should put more trust in larger scale organizations who are doing exploring, like GiveWell, and pool our resources.
  • Individuals should work harder to put relevant information about information we gather online.

 

I was partially mistaken in thinking about how to "prove" speculative causes.

I think there was some value in my essay "What Would It Take to Prove a Speculative Cause?" because it talked concretely about strategies some organizations could take to get more information about their impact.

But the overall concept is mistaken -- there is no arbitrary threshold of evidence at which a speculative cause needs to cross and I was wasting my time by trying to come up with one.  Instead, I think it's appropriate to continue doing expected value calculations as long as we maintain a self-skeptical, pro-measurement mindset.

 

I had previously not fully taken into account the cost of acquiring further information.

The important question in value of information is not "what does this information get me in terms of changing my beliefs and actions?" but actually "how valuable is this information?", as in, do the benefits of gathering this information outweigh all the costs?  In some cases, I think the benefits of further proving a cause probably don't outweigh the costs.

For one possibly extreme example, while I don't know the rationale for doing a 23rd randomized controlled trial on anti-malaria bednets after performing the previous 22, it's likely that doing that RCT would have to be testing something more specific than the general effectiveness of bednets to justify the high cost of doing an RCT.

Likewise, there are costs on organizations to devoting resources to measuring themselves and being more transparent.  I don't think these costs are particularly high or defeat the idea of devoting more resources to this area, but I hadn't really taken them into account before. 

 

I'm slightly more in favor of acting randomly (trial and error).

I still think it's difficult to acquire good value of information and it's very easy to get caught "spinning our wheels" in research, especially when that research has no clear feedback loops.  One example, perhaps somewhat controversial, would be to point to the multi-century lack of progress on some problems in philosophy (think meta-ethics) as an example of what can happen to a field when there aren't good feedback loops to ground yourself.

However, I underestimated the amount of information that comes forward just doing ones normal activities.  The implication here is that it's more worthwhile than I initially thought to fund speculative causes just to have them continue to scale and operate.

-

(This was also cross-posted on my blog.)

Comments (51)

Comment author: RyanCarey 16 August 2013 01:24:20PM 8 points [-]

An impressive post. On a personal note, sometimes I think that I will one day lose the ability to change my mind. I will become dull, stubborn and conservative, and keep publishing rephrasings of my same old views, as do many philosophers and academics. From then on, I will just keep rationalising my existing views, on and on, until I die. Whatever degree of neuroplasticity ageing people have, this probably motivates me to update aggressively, and now! So congratulations on being mentally alive!

Now, specific feedback: As per your initial post, you highly value the high future. As per this post, you favour exploration over exploitation.

Now that you have inreased your valuation of Givewell, you should do the same for the Future of Humanity Institute and Center for Study of Existential Risk, right? If you still do not value FHI and CSER, perhaps you still want to improve the future with interventions like AMF’s that have ‘proven’ near-future benefits. But what about scanning for asteroids? This has a pretty straightforward case for its benefits, but is not ‘proven’, and is never going to be supported by an RCT or cohort study. I’m not sure you’ve really thought through the consequences of dissolving this concept of ‘proven’ vs ‘unproven’. Your thoughts?

Also, I predict some pushback from LW on your use of the word 'random’. ‘Educated guess’ or ‘estimate’ seem better.

Lastly, this should be promoted to main, because it is high quality, and this rapid update should be presented beside the original contention.

Comment author: CarlShulman 16 August 2013 06:20:26PM *  10 points [-]

But what about scanning for asteroids?

That's mostly solved, all the dinosaur-killers have been tracked, and 90%+ of the 1 km size ones. So there's not much room for more funding left, and it seems very likely that's not the best thing to work on in terms of existential risk (the mopping-up effort is now increasingly aimed at city-smashers or tsunami-triggers).

Dark comet risk has been less addressed, because it is much harder to track such comets than asteroids.

Wikipedia:

It is currently (as of late 2007) believed that there are approximately 20,000 objects capable of crossing Earth's orbit and large enough (140 meters or larger) to warrant concern.[43] On the average, one of these will collide with Earth every 5,000 years, unless preventative measures are undertaken.[44] It is now anticipated that by year 2008, 90% of such objects that are 1 km or more in diameter will have been identified and will be monitored.

(This has happened).

The further task of identifying and monitoring all such objects of 140m or greater is expected to be complete around 2020.

Comment author: Lumifer 16 August 2013 07:06:21PM -1 points [-]

That's mostly solved

I don't think so. The asteroid problem doesn't involve only what the astronomers technically call asteroids. In involves any sufficiently large body moving at sufficiently high speed on the intercept trajectory.

To "mostly solve" this problem you need either to account for all sufficiently large bodies in the Solar system (we'll agree not to worry about whatever might arrive out of interstellar space) or build some kind of deflection/destruction system which can handle everything the Solar system can throw at our planet.

Comment author: gwern 16 August 2013 07:39:20PM 3 points [-]

To "mostly solve" this problem you need either to account for all sufficiently large bodies in the Solar system (we'll agree not to worry about whatever might arrive out of interstellar space)

You should probably read Carl's link.

Comment author: Lumifer 16 August 2013 07:46:25PM 0 points [-]

I did. I am not impressed by the statistics quoted in it. In particular, there is a neat trick in transitioning from "NASA reports that all near-earth asteroids larger than 10 kilometers in diameter ... have already been identified" to "This eliminates much of the estimated risk due to {note the glaring empty space here where words "near-earth" used to be} asteroids"

Asteroid impact is mostly a black-swan type of problem: you can identify the risk you see but you have very little idea of the remaining risk from things you do not see.

Comment author: JoshuaZ 16 August 2013 08:11:18PM 5 points [-]

Near earth asteroids are the primary threat set here, with only a tiny fraction not in that set that have any chance of hitting Earth. That's precisely why they say it eliminates much of the estimated risk due to asteroids.

Comment author: Lumifer 16 August 2013 08:31:09PM 0 points [-]

Near earth asteroids are the primary threat set here

I think it used to be the primary threat set. The claim is that near earth asteroids are not a threat because we looked at them and established that large ones are not going to hit Earth in the near future. Thus near earth asteroids are not the primary threat set any more.

Comment author: JoshuaZ 18 August 2013 07:15:56PM 1 point [-]

This seems like an odd use of language which misses the fundamental point: the observation of the near Earth asteroids reduces the estimated risk level by orders of magnitude. Whether the remaining risk is still concentrated in the near-earth case is a secondary consideration and not relevant to what was being discussed.

Comment author: Lumifer 19 August 2013 06:11:41PM 1 point [-]

I don't know why you think this use of language is odd. Saying that "we thought X was dangerous, we looked at it closely and it turns out X isn't dangerous at all" has the same meaning as "we mis-estimated the danger from X and then corrected the estimate".

If your updated belief is that there is little danger from near-earth asteroids, then the original belief that near-earth asteroids were the primary threat set was incorrect.

Comment author: JoshuaZ 19 August 2013 06:31:32PM 0 points [-]

Because it misses the point that the total risk was from asteroids isn't that high. Yes, of the remaining asteroid threat, more of it is from non near Earth asteroids, but that's not relevant to the discussion at hand. Hence the phrase in the report that you objected to ""This eliminates much of the estimated risk due to asteroids" makes complete sense.

Comment author: timtyler 21 August 2013 11:52:06PM *  -1 points [-]

Saying that "we thought X was dangerous, we looked at it closely and it turns out X isn't dangerous at all" has the same meaning as "we mis-estimated the danger from X and then corrected the estimate".

Risks and dangers here are percieved risks and dangers. In that context, such talk makes sense - obviously percieved risks depend on your current state of knowledge. Maybe god knows whether the bad thing will happen or not - but without a hotline to Him, percieved risks and dangers will remain the best we have.

Comment author: peter_hurford 18 August 2013 07:30:37AM 2 points [-]

I think that I will one day lose the ability to change my mind. I will become dull, stubborn and conservative, and keep publishing rephrasings of my same old views, as do many philosophers and academics.

I'm concerned about this too.

~

Now that you have increased your valuation of Givewell, you should do the same for the Future of Humanity Institute and Center for Study of Existential Risk, right? If you still do not value FHI and CSER, perhaps you still want to improve the future with interventions like AMF’s that have ‘proven’ near-future benefits.

I think this is less clear. I know a moderate amount about what GiveWell is making and how they intend to achieve and demonstrate progress in their domain. I think this is currently not true for FHI and CSER, but I could be convinced otherwise.

~

But what about scanning for asteroids? This has a pretty straightforward case for its benefits, but is not ‘proven’, and is never going to be supported by an RCT or cohort study.

I agree with the thoughts expressed by Carl Shulman and would have said the same thing. However, I do think it's possible there is scalable existential risk interventions that are reasonably understood and progress can be confidently made on them. I just don't yet know what they are.

For the record, my position is not "I need an RCT or I won't trust it".

Comment author: Crux 18 August 2013 09:09:38AM 1 point [-]

Certainly it seems to be the case that people have a lot harder of a time changing their mind once they get past their 40s or 50s, but at the same time I have met a few people closing in on their 70s who have made radical changes to their lives and their thinking. It's not impossible; just way more difficult.

Comment author: wiserd 16 August 2013 07:20:55PM *  3 points [-]

"We should put more trust in larger scale organizations who are doing exploring, like GiveWell, and pool our resources."

I wonder about the notion of scalability, because with any information provider there's the possible issue of corruption. Is it better to have a large organization which is easy to track? (Goodwill and the Red Cross seem to give their CEOs pretty high salaries, considering that they're charities.) Or is it better to have a system which is more robust to corruption and tunnel vision due to multiple redundancies? Do larger organizations attract a certain type of social climber? I have no doubt that there are economies of scale. Part of the reason I'm asking these questions is that corruption seems to be a universal human stumbling block, and one that is often inadequately addressed. We project generously and assume that the heads of organizations are "basically like us" and altruistic in their intent. Neither is probably true, as the upper ranks of large organizations tend to collect those who are particularly attracted to climbing the ranks.

Comment author: Eliezer_Yudkowsky 18 August 2013 03:41:36AM 5 points [-]

Upvoted for changing one's mind in public.

Comment author: cousin_it 16 August 2013 03:54:56PM *  2 points [-]

Interesting! What's the most cost-effective way to become more certain about the impact of something like MIRI?

Comment author: peter_hurford 16 August 2013 06:44:34PM 3 points [-]

I don't know nearly enough about MIRI or AI to say. My initial guess would be to get more external review from experts, but I imagine there's reasons why that hasn't happened much yet (though see Luke's and XiXiDu's interview series).

Comment author: lukeprog 17 August 2013 03:32:25AM 0 points [-]

Yes, I think external review by AI experts would be good, though we'd need to find people who show the ability to change their mind in response to evidence, and people who are willing to actually listen to and work through the arguments with us. I say more about this here.

Comment author: bentarm 17 August 2013 07:01:01AM *  7 points [-]

I imagine this isn't your intention, but this does read a lot like "I think external review like AI experts would be good, but if we do that review, and don't liek the results, it's because we picked the wrong AI experts."

Comment author: lukeprog 18 August 2013 12:07:25AM *  8 points [-]

Here are two things we want from an external review:

  1. MIRI wants to learn more about whether its basic assumptions and their apparent strategic implications are sound.
  2. Outsiders of various kinds want to learn more about whether MIRI is a reasonable thing to support and emulate.

These two groups (MIRI & outsiders) have wildly different information about risks from AI, and are thus in different positions with respect to how they should respond to external reviews of MIRI's core assumptions.

To illustrate the point, consider two actual events of direct or indirect "external review" of MIRI's core assumptions.

#1: Someone as intelligent and broadly informed as Russ Roberts was apparently relieved of some of his worry about AI risk upon hearing Kevin Kelly's reasoning on the matter. Given my state of information, I can immediately see that Kelly's point about the Church-Turing thesis isn't relevant to whether we should worry about AI (Ctrl+F on this page for "The basis of my non-worry"). But maybe Russ Roberts isn't in a position to do so, because he knows so little about AI and theory of computation that he can't tell whether Kelly's stated reasons for complacency are sound or not. So the result of this indirect "external review" of MIRI's assumptions is that Roberts has to say to himself "Well, I don't know what to think. Robin Hanson says we should worry but Kevin Kelly says we shouldn't, and they both have fairly good normal credentials on the issue of long-term tech forecasting relative to almost everyone." And I, meanwhile, can see that I haven't been presented with much reason to re-evaluate my core assumptions about AI risk.

#2: In mid-2012, Paul Christiano, who at the time had just finished his undergraduate degree, gave (w/ help from Carl Shulman) such a detailed, informed, and reasonable critique of Eliezer's standard case for high confidence in hard takeoff that both Eliezer and I have since mentally re-tagged "AI takeoff speed" as an "open question" rather than a "moderately closed question," prompting Eliezer to write up intelligence explosion microeconomics as his "Friendly AI Open Problem #1." Eliezer and I were in a position to make a fairly significant update from Paul's argument, but unfortunately Paul has far fewer normal credentials as an "expert worth listening to" than Kevin Kelly or Robin Hanson do, so outsiders probably aren't in a position to be moved much either way by Paul stating his opinion and his reasons on the issue, due to (fairly appropriate) epistemic learned helplessness.

Also note that if 10 of the world's top AI experts spent two weeks with MIRI and FHI trying to understand our arguments, and their conclusion was that AI wasn't much of a risk, and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we're on the right track. (I don't expect this to actually happen; this is just another illustration of the situation we're in with regard to external review.)

Related: Contrarian Excuses.

ETA: Also, I should mention that I don't gain any confidence in MIRI's core assumptions form the fact that people like Kevin Kelly or these people give bad arguments against the plausibility of intelligence explosion. As far as I can tell, these people aren't engaging the arguments in much detail, and are mostly just throwing up whatever rejections first occur to them. I would expect them to do that whether MIRI's core assumptions were correct or not. That's why I stipulated that to gain confidence in MIRI's core assumptions, we'd need to get a bunch of smart, reasonable people to investigate the arguments in detail, and try hard to extract good objections from them, and learn that they can't come up with good objections even under those circumstances.

Comment author: lukeprog 28 August 2013 11:01:34PM 4 points [-]

I'll also mention that GiveWell — clearly not a den of MIRI-sympathizers — is effectively doing an external review of some of MIRI's core assumptions, by way of "shallow investigations" of specific catastrophic risks (among other interventions).

So far, they've reviewed climate change, asteroids, supervolcanoes, and nuclear war. None of these reviews cite the corresponding chapters in GCR, perhaps because GiveWell wants to do its own review of these issues that is mostly independent from the Bostromian school (which includes MIRI).

So far, GiveWell seems to have come to the same conclusions as the Bostromian school has about these specific risks, with the major caveat that these are shallow investigations that are "not researched and vetted to the same level as [Givewell's] standard recommendations."

I'm pretty excited about GiveWell investigating GCRs independently of the Bostromian school, since (1) I admire the quality and self-skepticism of GiveWell's research so far, (2) I think GCRs are important to study, and (3) this will provide a pretty solid external review of some of MIRI's core assumptions about the severity of various x-risks.

Comment author: CarlShulman 18 August 2013 02:35:42AM 4 points [-]

Also note that if 10 of the world's top AI experts spent two weeks with MIRI and FHI trying to understand our arguments, and their conclusion was that AI wasn't much of a risk, and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we're on the right track.

Based on what we already know this would require a very unrepresentative sample, and cause wider revisions. And if they published such obviously unconvincing reasons it would lead to similar updates in many casual observers.

And so what we are going to do is, there is really almost no reason to make human-like intelligence because we can do it so easily in 9 months. Untrained workforce.

Yes, this argument is remarkably unconvincing. Human labor is still costly, limited in supply (it's not 9 months, it's 20+ years, with feeding, energy costs, unreliable quality and many other restrictions), and so forth.

Comment author: private_messaging 27 August 2013 10:41:53PM 0 points [-]

Based on what we already know this would require a very unrepresentative sample.

There's too much focus on confirmation - e.g. if it is false, there must be some update in the opposite direction, but in practice one would just say that "those top 10 AI experts took us seriously and engaged out arguments, which boosts our confidence that we are on the right track".

Comment author: private_messaging 27 August 2013 12:59:07PM 2 points [-]

and their primary reasons were (1) a denial of causal functionalism and (2) the Chinese room argument, and we had tried hard to elicit other objections, then MIRI and FHI should update in favor of more confidence that we're on the right track.

Unpack "top 10 AI experts" as "who we think the top 10 AI experts are" and unpack "their primary reasons were" with "we think their primary reasons were", and this updating will sound a lot more silly, especially if further conditioned by "we do not have a way to show our superiority in a more objective manner".

Comment author: ESRogs 06 September 2013 07:51:10PM 1 point [-]

Is Paul Christiano's hard takeoff critique publicly available?

Comment author: lukeprog 06 September 2013 07:54:53PM 2 points [-]

No. It was given over a series of in-person conversations and whiteboard calculations, with only scattered notes taken throughout. Paul does describe his own "mainline AI scenario" here, though.

Comment author: ESRogs 06 September 2013 09:05:43PM 0 points [-]

Ah, thanks.

Comment author: peter_hurford 18 August 2013 07:27:08AM 1 point [-]

MIRI wants to learn more about whether its basic assumptions and their apparent strategic implications are sound.

I still think it would be valuable to hear what relevant, independent AI experts think about these basic assumptions and strategic implications, perhaps accompanied with a detailed theory as to why they've come to wrong answers and MIRI has more advanced insight.

Comment author: lukeprog 28 August 2013 10:49:46PM 0 points [-]

Well, we can do this for lots of specific cases. E.g. last time I spoke to Peter Norvig, he said his reason for not thinking much about AI risk at this point (despite including a discussion of it in his AI textbook) was that he's fairly confident AI is hundreds of years away. Unfortunately, I didn't have time to walk him through the points of When Will AI Be Created? to see exactly why we disagreed on this point.

This will all be a lot easier when Bostrom's Superintelligence book comes out next year, so that experts can reply to the basic theses of our view when they are organized neatly in one place and explained in some detail with proper references and so on.

Comment author: John_Maxwell_IV 11 June 2015 03:09:01PM 0 points [-]

We should put more trust in larger scale organizations who are doing exploring, like GiveWell, and pool our resources.

I think at a certain point it's better to have several medium-sized organizations rather than a single large one. When my dad was thinking about graduate school, his professors encouraged him to go to a different university than the one he was attending in order to get a different perspective. I think the ideal case for the EA movement is to have multiple charity evaluators that share information with one another but operate independently, in the same way universities share information with one another and collaborate on projects while still operating independently. Also, it seems like organizations become more likely to degrade as they grow.