Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

How does MIRI Know it Has a Medium Probability of Success?

19 Post author: peter_hurford 01 August 2013 11:42AM

In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success.  What is this probability estimate based on and how is success defined?

I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.

-

(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)

 

Comments (146)

Comment author: CarlShulman 01 August 2013 07:15:05PM *  19 points [-]

People differ in their estimates within MIRI. Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization's future (and a high probability that the future will be very powerfully shaped by artificial intelligence in a way that can be affected by initial conditions). But the impact of MIRI in particular has to be a far smaller subset of the expected impact of the cause as a whole, in light of its limited scale and capabilities relative to the relevant universes (total AI research, governments, etc), the probability that AI is not close enough for MIRI to be very relevant, the probability that MIRI's approach turns out irrelevant, uncertainty over the sign of effects due to contributions to AI progress, future AI risk efforts/replaceability, and various other drag factors.

ETA: To be clear, I think that MIRI's existence, relative to the counterfactual in which it never existed, has been a good thing and reduced x-risk in my opinion, despite not averting a "medium probability," e.g. 10%, of x-risk.

ETA2: Probabilities matter because there are alternative uses of donations and human capital.

I have just spent a month in England interacting extensively with the EA movement here. Donors concerned with impact on the long-run future are considering donations to all of the following (all of these are from talks with actual people making concrete short-term choices; in addition to donations, people are also considering career choices post-university):

  • 80,000 Hours, the Center for Effective Altruism and other organizations that are helping altruists to improve their careers, coordination, information, and do movement building; some specifically mention the Center For Applied Rationality; these organizations also improve non-charity options, e.g. 80k helping people going into scientific funding agencies and political careers where they will be in a position to affect research and policy reactions to technologies relevant to x-risk and other trajectory changes
  • AMF/GiveWell's other recommended charities to keep GiveWell and the EA movement growing (GiveWell's growth in particular has been meteoric, with less extreme but still rapid growth in other EA institutions such as Giving What We can and CEA), while actors like GiveWell Labs, Paul Christiano, and Nick Beckstead and others at FHI, investigate the intervention options and cause prioritization, followed by organization-by-organization analysis of the GiveWell variety, laying the groundwork for massive support for the interventions and organizations identified by such processes as most effective in terms of their far future impact
  • Finding ways to fund such evaluation with RFMF, e.g. by paying for FHI or CEA hires to work on them
  • The FHI's other work
  • A donor-advised fund investing the returns until such evaluations or more promising opportunities present themselves or are elicited by the fund, including both known options for which no organization with RFMF or adequate quality exists, and unknown future options; some possible applications include, e.g. convening panels of independent scientific experts to evaluate key technical claims about future technologies, extensions of the DAGGRE forecasting methods, a Bayesian aggregation algorithm that greatly improves extraction of scientific expert opinion or science courts that could mobilize much more talent and resources to neglected problems with good cases, some key steps in biotech enhancement, AI safety research when AI is better understood, and more

This Paul Christiano post discusses the virtues of the donor-advised fund/"Fund for the Future" approach; Giving What We Can has already set up a charitable trust to act as a donor-advised fund in the UK, with one coming soon in the US, and Fidelity already offers a standardized donor-advised fund in America (DAFs allow one to claim tax benefits of donation immediately and then allow the donation to compound); there was much discussion this month about the details of setting up a DAF dedicated to far future causes (the main logistical difficulties are setting up the decision criteria, credibility, and maximum protection from taxation and disruption)

Comment author: lukeprog 07 August 2013 09:14:15PM *  9 points [-]

I'm eager to see Eliezer's planned reply to your "ETA2", but in the meantime, here are a few of my own thoughts on this...

My guess is that movement-building and learning are still the best things to do right now for AI risk reduction. CEA, CFAR, and GiveWell are doing good movement-building, though the GiveWell crowd tends to be less interested in x-risk mitigation. GiveWell is doing a large share of the EA-learning, and might eventually (via GiveWell labs) do some of the x-risk learning (right now GiveWell has a lot of catching up to do on x-risk).

The largest share of the "explicit" x-risk learning is happening at or near FHI & MIRI, including e.g. Christiano. Lots of "implicit" x-risk learning is happening at e.g. NASA where it's not clear that EA-sourced funding can have much marginal effect relative to the effect it could have on tiny organizations like MIRI and FHI.

My impression, which could be wrong, is that GiveWell's ability to hire more researchers is not funding-limited but rather limited by management's preference to offer lower salaries than necessary to ensure cause loyalty. (I would prefer GiveWell raise salaries and grow its research staff faster.) AMF could be fully funded relatively easily by Good Ventures or the Gates Foundation but maybe they're holding back because this would be discouraging to the EA movement: small-scale donors requiring the high-evidence threshold met by GiveWell's top charities would say "Well, I guess there's nothing for little 'ol me to do here." (There are other reasons they may be holding back, too.)

I think accelerating learning is more important right now than a DAF. Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we've learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goodness of particular x-risk mitigation efforts, the sooner large funders will fund those efforts. Or, as Christiano writes:

To me it currently looks like the value of getting information faster is significantly higher than the value of money, and on the current margin I think most of these learning activities are underfunded.

And:

A relatively small set of activities seems to be responsible for most learning that is occurring (for example, much of GiveWell’s work, some work within the Centre for Effective Altruism, some strategy work within MIRI, hopefully parts of this blog, and a great number of other activities that can’t be so easily sliced up)

However, Paul thinks there are serious RFMF problems here:

A more serious concern is that there seems to currently be a significant deficit of human capital specialized for this problem and willing to work on it (without already being committed to work on it), so barring some new recruitment strategies (e.g. paying market wages for non-EAs to do EA strategy research) there are significant issues with room for more funding.

In contrast, I think there is plenty of room for more funding here, even without resorting to "paying market wages for non-EAs to do EA strategy research":

  1. MIRI could run more workshops and hire some able and willing FAI researchers, which I think is quite valuable for x-risk mitigation strategy learning apart from the object-level FAI progress it might produce. But even excluding this...

  2. With more cash, FHI and CSER could host strategy-relevant conferences and workshops, and get people like Stuart Russell and Richard Posner to participate.

  3. I have plenty of EAs capable of doing the labor-intensive data-gathering work needed for much of the strategy work, e.g. collecting data on how fast different parts of AI are progressing, how much money has gone into AI R&D each decade since the 60s, how ripple effects have worked historically, more IEM-relevant data like Katja's tech report, etc. I just don't have the money to pay them to do it.

  4. FHI has lots more researcher-hours it could purchase if it had more cash.

Finally, a clarification: If I think movement-building and learning are most important right now, why is MIRI focused on math research this year? My views on this have shifted even since our 2013 strategy post, and I should note that Eliezer's reasons for focusing on math research are probably somewhat different from mine.

In my estimation, MIRI's focus on math research offers the following benefits to movement-building and learning:

  1. Math research has better traction than strategic research with the world's top cognitive ability. And once top talent is engaged by the math research, some of these top thinkers turn their attention to the strategic issues, too. (Historically true, not just speculation.)

  2. Without an object-level research program on the most important problem (beneficent superintelligence), many of the best people just "bounce off" because there's nothing for them to engage directly. (Historically true, not just speculation.)

  3. And of course, FAI research tells us some things about how hard FAI research is, which lines of inquiry are tractable now, etc.

Comment author: ciphergoth 09 August 2013 12:08:40PM 2 points [-]

Your reasons for focusing on math research at MIRI seem sound, but I take it you've noticed the warning sign of finding that what you already decided to do turns out to be a good idea for different reasons than you originally thought?

Comment author: lukeprog 09 August 2013 07:10:40PM 1 point [-]

Yes, though these reasons are pretty similar to the reasons that made me switch positions on strategy back when I thought a focus on strategic research would be best for MIRI in 2013.

Comment author: CarlShulman 07 August 2013 10:42:54PM 2 points [-]

My impression, which could be wrong, is that GiveWell's ability to hire more researchers is not funding-limited but rather limited by management's preference to offer lower salaries than necessary to ensure cause loyalty.

It's clearly not funding-limited, as they have plenty of funding for operations. I'm less confident of the salaries explanation as to their difficulties hiring: it is quite plausible as a significant factor.

And the "give more money via GiveWell's top charities to accelerate their approach to their limits of growth" rationale gets worse every year, as the contribution of a dollar to their money moved falls by half, and the number of potential doublings remaining falls. So I would not see direct donations to GiveWell or its top charities as competitive, although other interventions that bolstered it more effectively could.

Two plausible examples: 80,000 hours might deliver a number of good new hires to GiveWell, or the Effective Fundraising experimental project, inspired by discussion like this 80,000 hours blog post, may succeed in efficiently mobilizing non-EA funds to support GiveWell's top charities.

Getting high-quality evidence about which x-risk mitigation efforts are worthwhile requires lots of work, but one thing we've learned in the past decade is that causes with high-quality evidence for their effectiveness tend to get funded, and this trend is probably increasing. The sooner we do enough learning to have high-quality evidence for the goodness of particular x-risk mitigation efforts, the sooner large funders will fund those efforts.

Yes.

I think accelerating learning is more important right now than a DAF.

One of the biggest virtues of a large "fund for the future," IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style.

And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area).

In contrast, I think there is plenty of room for more funding here, even without resorting to "paying market wages for non-EAs to do EA strategy research...[FHI RFMF to hire more academics/free up grant-related time from existing staff...MIRI math workshops]...[hiring people to collect and publish data on past AI predictions, past AI progress, past inputs into the AI field, etc...]

Good points that I'm largely on board with, qualitatively, although one needs to make more of a case to show they meet the bar of beating existing alternatives, or waiting for others to enter the field and do things better.

Also, I should mention the Global Catastrophic Risks Institute, even if no one at the EA events in England mentioned it while I was there.

Comment author: lukeprog 09 August 2013 07:57:37PM 0 points [-]

One of the biggest virtues of a large "fund for the future," IMHO, is that it would make it easier to start up new projects in the field separate from existing organizations if they could meet the (transparently announced) standards of the fund, with the process as transparent and productive of information as practicable, GiveWell style.

And it could serve those who think the existing organizations in the field are deficient in some remediable way (rather than having some general objection to all work in the area).

That does sound good. Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF? Are there such standards in place for the one in the UK?

Comment author: CarlShulman 09 August 2013 08:41:45PM 0 points [-]

The one in the UK is mainly functioning as a short term DAF along the lines of "direct the money as you intend, with trust of GWWC as a backstop" which is fine if you don't want to delay disbursement until after you die.

Is there any ongoing progress on figuring out what those transparently announced standards could be, and how one might set up such a DAF?

Not yet, so far mainly discussions, e.g. with Paul, Rob Wiblin, Nick Beckstead, et al. I expect more from CEA on this (not wholly independently of my own actions).

Comment author: peter_hurford 02 August 2013 07:27:16AM 2 points [-]

Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

Are these available? Are they the standard stuff (i.e., "Evidence and Import")?

~

For myself, I think the cause of AI risk reduction, in total and over time, has a worthwhile small-to-medium probability of making an astronomical difference on our civilization's future

How do you arrive at that conclusion? I'm less skeptical of the cause-specific claim than the organization-specific claim, but it's worth digging deeper into.

Comment author: CarlShulman 02 August 2013 12:54:06PM 4 points [-]

Eliezer...Are these available? Are they the standard stuff (i.e., "Evidence and Import")?

Yes, and his posts about intelligence explosion on Overcoming Bias, this, this, and unfortunately comments scattered around Less Wrong or various interveiws that would take some work to find and gather in one place.

How do you arrive at that conclusion? I'm less skeptical of the cause-specific claim than the organization-specific claim, but it's worth digging deeper into.

Nick Bostrom's book on superintelligence probably provides the best single treatment now, having synthesized most pre-existing work. It is moving towards publication, but you might ask him if you can read the draft.

Comment author: lukeprog 07 August 2013 10:00:32PM 1 point [-]

having synthesized most pre-existing work

Most pre-existing work? I would've said "having synthesized ~5% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI."

Comment author: lukeprog 06 October 2013 07:49:32PM 1 point [-]

Good news! Having now read the near-finished draft, my new guess is that Bostrom's book synthesizes more like 20% of pre-existing work related to superintelligence strategy that has been done at or near MIRI and FHI. A lot has been added to the book since April. It's really killing me that the book won't be published until mid 2014.

Comment author: CarlShulman 07 August 2013 10:10:51PM 0 points [-]

One can delve indefinitely into any subtopic, but with diminishing returns. Do you think that it doesn't address most of the higher-level topic areas, if not all of the issues arising therein?

Comment author: lukeprog 07 August 2013 10:41:13PM 0 points [-]

No, I think it does a pretty good job of that. I'm not arguing that the book should be different than it is. I'm just saying that it definitely doesn't synthesize "most" pre-existing work.

Comment author: Pablo_Stafforini 25 February 2017 10:17:05PM 1 point [-]

Eliezer has not published a detailed explanation of his estimates, although he has published many of his arguments for his estimates.

Eliezer wrote this in 1999:

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015.

Comment author: gjm 26 February 2017 02:46:02AM 4 points [-]

Hasn't Eliezer said, on every occasion since the beginning of LW when the opportunity has arisen, that Eliezer-in-1999 was disastrously wrong and confused about lots of important things?

(I don't know whether present-day-Eliezer thinks 18-years-ago-Eliezer was wrong about this particular thing, but I would be cautious about taking things he said that long ago as strongly indicative of his present opinions.)

Comment author: Pablo_Stafforini 26 February 2017 08:46:45PM *  1 point [-]

Yes, I am aware that this is what Eliezer has said, and I wasn't implying that those early statements reflect Eliezer's current thinking. There is a clear difference between "Eliezer believed this in the past, so he must believe it at present" and "Eliezer made some wrong predictions in the past, so we must treat his current predictions with caution". Eliezer is entitled to ask his readers not to assume that his past beliefs reflect those of his present self, but he is not entitled to ask them not to hold him responsible for having once said stuff that some may think was ill-judged. (If Eliezer had committed serious crimes at the age of 18, it would be absurd for him to now claim that we should regard that person as a different individual who also happens to be called 'Eliezer Yudkowsky'. Epistemic responsibility seems analogous to moral responsibility in this respect.)

Comment author: gjm 26 February 2017 09:03:31PM 1 point [-]

I hadn't realised anyone was arguing for not treating Eliezer's current predictions with caution. I can't imagine why anyone wouldn't treat anyone's predictions with caution in this field.

Comment author: Pablo_Stafforini 26 February 2017 09:09:25PM *  1 point [-]

My point is that these early pronouncements are (limited) evidence that we should treat Eliezer's predictions with more caution that we would otherwise.

Comment author: gjm 28 February 2017 11:59:24AM 0 points [-]

OK, I guess. I have to say that the main impression I'm getting from this exchange is that you wanted to say "boo Eliezer"; it seems like if you wanted to make an actual usefully constructive point you'd have been somewhat more explicit in your original comment. ("Eliezer wrote this in 1999: [...]. I know that Eliezer has since repudiated a lot of his opinions and thought processes of that period, but if his opinions were that badly wrong in 1999 then we shouldn't take them too seriously now either." or whatever.)

I will vigorously defend anyone's right to say "boo Eliezer" or "yay Eliezer", but don't have much optimism about getting a useful outcome from a conversation that begins that way, and will accordingly drop it now.

Comment author: Pablo_Stafforini 03 March 2017 06:52:07AM 0 points [-]

Thanks for the feedback. I agree that a comment worded in the manner you suggest would have communicated my point more effectively.

Comment author: Houshalter 26 February 2017 02:04:57AM 2 points [-]

Yudkowsky has changed his views a lot over the last 18 years though. A lot of his earlier writing is extremely optimistic about AI and it's timeline.

Comment author: bogus 26 February 2017 12:45:16AM *  0 points [-]

Well, a nanowar is just a conflict on a very, very small scale - like many orders of magnitude less serious than your average barfight. Perhaps we had one before 2015 and nobody noticed! Now we just have to wait until 2020 for the seed AI to transcend.

Comment author: Eliezer_Yudkowsky 02 August 2013 08:51:38AM 24 points [-]

Most fundamentally, it's based on taking at face value a world in which nobody appears to be doing similar work or care sufficiently to do so. In the world taken at face value, MIRI is the only organization running MIRI's workshops and trying to figure out things like tiling self-modifying agents and getting work started early on what is probably a highly serial time-sensitive task.

Success is defined most obviously as actually constructing an FAI, and it would be very dangerous to have any organizational model in which we were not trying to do this (someone who conceives of themselves as an ethicist whose duty it is to lecture others, and does not intend to solve the problem themselves, is exceedingly unlikely to confront the hardest problems). But of course if our work were picked up elsewhere and reused after MIRI itself died as an organization for whatever reason, or if in any general sense the true history as written in the further future says that MIRI mattered, I should not count my life wasted, nor feel that we had let down MIRI's donors.

Comment author: Furcas 02 August 2013 02:55:33PM *  9 points [-]

Ok, but that doesn't increase the probability to 'medium' from the very low initial probability of MIRI or another organization benefiting from MIRI's work solving the extremely hard problem of Friendly AI before anyone else screws it up.

I've read all your posts in the threads linked by the OP, and if multiplying the high beneficial impact of Friendly AI by the low probability of success isn't allowed, I honestly don't see why I should donate to MIRI.

Comment author: Eliezer_Yudkowsky 02 August 2013 05:42:05PM 13 points [-]

If this was a regular math problem and it wasn't world-shakingly important, why wouldn't you expect that funding workshops and then researchers would cause progress on it?

Assigning a very low probability to progress rests on a sort of backwards reasoning wherein you expect it to be difficult to do things because they are important. The universe contains no such rule. They're just things.

It's hard to add a significant marginal fractional pull to a rope that many other people are pulling on. But this is not a well-tugged rope!

Comment author: Furcas 02 August 2013 07:19:58PM *  10 points [-]

I'm not assigning a low probability to progress, I'm assigning a low probability to success.

Where FAI research is concerned, progress is only relevant in as much as it increases the probability of success, right?

Unlike a regular math problem, you've only got one shot at getting it right, and you're in a race with other researchers who are working on an easier problem (seed AI, Friendly or not). It doesn't matter if you're 80% of the way there if we all die first.

Edited to add and clarify: Even accounting for the progress I think you're likely to make, the probability of success remains low, and that's what I care about.

Comment author: Eliezer_Yudkowsky 02 August 2013 08:56:39PM 6 points [-]

Clarifying question: What do you think is MIRI's probability of having been valuable, conditioned on a nice intergalactic future being true?

Comment author: Furcas 02 August 2013 09:12:42PM 1 point [-]

Pretty high. More than 10%, definitely. Maybe 50%?

Comment author: CarlShulman 02 August 2013 09:56:33PM *  13 points [-]

A non-exhaustive list of some reasons why I strongly disagree with this combination of views:

  • AI which is not vastly superhuman can be restrained from crime, because humans can be so restrained, and with AI designers have the benefits of the ability to alter the mind's parameters (desires, intuitions, capability for action, duration of extended thought, etc) inhibitions, test copies in detail, read out its internal states, and so on, making the problem vastly easier (although control may need to be tight if one is holding back an intelligence explosion while this is going on)
  • If 10-50 humans can solve AI safety (and build AGI!) in less than 50 years, then 100-500 not very superhuman AIs at 1200x speedup should be able to do so in less than a month
  • There are a variety of mechanisms by which humans could monitor, test, and verify the work conducted by such systems
  • The AIs can also work on incremental improvements to the control mechanisms being used initially, with steady progress allowing greater AI capabilities to develop better safety measures, until one approaches perfect safety
  • If a small group can solve all the relevant problems over a few decades, then probably a large portion of the AI community (and beyond) can solve the problems in a fraction of the time if mobilized
  • As AI becomes visibly closer such mobilization becomes more likely
  • Developments in other fields may make things much easier: better forecasting, cognitive enhancement, global governance, brain emulations coming first, global peace/governance
  • The broad shape of AI risk is known and considered much more widely than MIRI: people like Bill Gates and Peter Norvig consider it, but think that acting on it now is premature; if they saw AGI as close, or were creating it themselves, they would attend to the control problems
Comment author: Wei_Dai 03 August 2013 11:40:55AM 9 points [-]

Paul Christiano, and now you, have started using the phrase "AI control problems". I've gone along with it in my discussions with Paul, but before many people start adopting it maybe we ought to talk about whether it makes sense to frame the problem that way (as opposed to "Friendly AI"). I see a number of problems with it:

  1. Control != Safe or Friendly. An AI can be perfectly controlled by a human and be extremely dangerous, because most humans aren't very altruistic or rational.
  2. The framing implicitly suggests (and you also explicitly suggest) that the control problem can be solved incrementally. But I think we have reason to believe this is not the case, that in short "safety for superintelligent AIs" = "solving philosophy/metaphilosophy" which can't be done by "incremental improvements to the control mechanisms being used initially".
  3. "Control" suggests that the problem falls in the realm of engineering (i.e., belongs to the reference class of "control problems" in engineering, such as "aircraft flight control"), whereas, again, I think the real problem is one of philosophy (plus lots of engineering as well of course, but philosophy is where most of the difficulty lies). This makes a big difference in trying to predict the success of various potential attempts to solve the problem, and I'm concerned that people will underestimate the difficulty of the problem or overestimate the degree to which it's parallelizable or generally amenable to scaling with financial/human resources, if the problem becomes known as "AI control".

Do you disagree with this, on either the terminological issue ("AI control" suggests "incremental engineering problem") or the substantive issue (the actual problem we face is more like philosophy than engineering)? If the latter, I'm surprised not to have seen you talk about your views on this topic earlier, unless you did and I missed it?

Comment author: CarlShulman 03 August 2013 12:59:58PM 3 points [-]

Thanks for those thoughts.

Nick Bostrom uses the term in his book, and it's convenient for separating out pre-existing problems with "we don't know what to do with our society long term, nor is it engineered to achieve that" and the particular issues raised by AI.

But I think we have reason to believe this is not the case, that in short "safety for superintelligent AIs" = "solving philosophy/metaphilosophy" which can't be done by "incremental improvements to the control mechanisms being used initially".

In the situation I mentioned, not vastly superintelligent initially (and capabilities can vary along multiple dimensions, e.g. one can have many compartmentalized copies of an AI system that collectively deliver a huge number of worker-years without any one of them possessing extraordinary capabilities.

What is your take on the strategy-swallowing point: if humans can do it, then not very superintelligent AIs can.

"Control" suggests that the problem falls in the realm of engineering (i.e., belongs to the reference class of "control problems" in engineering, such as "aircraft flight control")...I'm concerned that people will underestimate the difficulty of the problem or overestimate the degree to which it's parallelizable or generally amenable to scaling with financial/human resources, if the problem becomes known as "AI control".

There is an ambiguity there. I'll mention it to Nick. But, e.g. Friendliness just sounds silly. I use "safe" too, but safety can be achieved just by limiting capabilities, which doesn't reflect the desire to realize the benefits.

Comment author: Wei_Dai 03 August 2013 04:38:44PM 6 points [-]

What is your take on the strategy-swallowing point: if humans can do it, then not very superintelligent AIs can.

It's easy to imagine AIXI-like Bayesian EU maximizers that are powerful optimizers but incapable of solving philosophical problems like consciousness, decision theory, and foundations of mathematics, which seem to be necessary in order to build an FAI. It's possible that that's wrong, that one can't actually get to "not very superintelligent AIs" unless they possessed the same level of philosophical ability that humans have, but it certainly doesn't seem safe to assume this.

BTW, what does "strategy-swallowing" mean? Just "strategically relevant", or more than that?

But, e.g. Friendliness just sounds silly. I use "safe" too, but safety can be achieved just by limiting capabilities, which doesn't reflect the desire to realize the benefits.

I suggested "optimal AI" to Luke earlier, but he didn't like that. Here are some more options to replace "Friendly AI" with: human-optimal AI, normative AI (rename what I called "normative AI" in this post to something else), AI normativity. It would be interesting and useful to know what options Eliezer considered and discarded before settling on "Friendly AI", and what options Nick considered and discarded before settling on "AI control".

(I wonder why Nick doesn't like to blog. It seems like he'd want to run at least some of the more novel or potentially controversial ideas in his book by a wider audience, before committing them permanently to print.)

Comment author: torekp 04 August 2013 03:20:19AM 1 point [-]

it's convenient for separating out pre-existing problems with "we don't know what to do with our society long term, nor is it engineered to achieve that" and the particular issues raised by AI.

I don't think that separation is a good idea. Not knowing what to do with our society long term is a relatively tolerable problem until an upcoming change raises a significant prospect of locking-in some particular vision of society's future. (Wei-Dai raises similar points in your exchange of replies, but I thought this framing might still be helpful.)

Comment author: Vladimir_Nesov 03 August 2013 12:38:16PM *  1 point [-]

If we are talking about goal definition evaluating AI (and Paul was probably thinking in the context of some sort of indirect normativity), "control" seems like a reasonable fit. The primary philosophical issue for that part of the problem is decision theory.

(I agree that it's a bad term for referring to FAI itself, if we don't presuppose a method of solution that is not Friendliness-specific.)

Comment author: Kawoomba 03 August 2013 08:29:52AM 5 points [-]

What do you think is MIRI's probability of having been valuable, conditioned on a nice intergalactic future being true?

  • More than 10%, definitely. Maybe 50%?

A non-exhaustive list of some reasons why I strongly disagree with this combination of views

Not that it should be used to dismiss any of your arguments, but reading your other comments in this thread I thought you must be playing devil's advocate. Your phrasing here seems to preclude that possibility.

If you are so strongly convinced that while AGI is a non-negligible x-risk, MIRI will probably turn out to have been without value even if a good AGI outcome were to be eventually achieved, why are you a research fellow there?

I'm puzzled. Let's consider an edge case: even if MIRI's factual research turned out to be strictly non-contributing to an eventual solution, there's no reasonable doubt that it has raised awareness of the issue significantly (in relative terms).

Would the current situation with the CSER or FHI be unchanged or better if MIRI had never existed? Do you think those have a good chance of being valuable in bringing about a good outcome? Answering 'no' to the former and 'yes' to the latter would transitively imply that MIRI is valuable as well.

I.e. that alone --nevermind actual research contributions -- would make it valuable in hindsight, given an eventual positive outcome. Yet you're strongly opposed to that view?

Comment author: CarlShulman 03 August 2013 01:17:26PM *  7 points [-]

The "combination of views" includes both high probability of doom, and quite high probability of MIRI making the counterfactual difference given survival. The points I listed address both.

If you are so strongly convinced that while AGI is a non-negligible x-risk, MIRI will probably turn out to have been without value even if a good AGI outcome were to be eventually achieved, why are you a research fellow there?

I think MIRI's expected impact is positive and worthwhile. I'm glad that it exists, and that it and Eliezer specifically have made the contributions they have relative to a world in which they never existed. A small share of the value of the AI safety cause can be quite great. That is quite consistent with thinking that "medium probability" is a big overestimate for MIRI making the counterfactual difference, or that civilization is almost certainly doomed from AI risk otherwise.

Lots of interventions are worthwhile even if a given organization working on them is unlikely to make the counterfactual difference. Most research labs working on malaria vaccines won't invent one, most political activists won't achieve big increases in foreign aid or immigration levels or swing an election, most counterproliferation expenditures won't avert nuclear war, asteroid tracking was known ex ante to be far more likely to discover we were safe than that there was an asteroid on its way and ready to be stopped by a space mission.

The threshold for an x-risk charity of moderate scale to be worth funding is not a 10% chance of literally counterfactually saving the world from existential catastrophe. Annual world GDP is $80,000,000,000,000, and wealth including human capital and the like will be in the quadrillions of dollars. A 10% chance of averting x-risk would be worth trillions of present dollars.

We've spent tens of billions of dollars on nuclear and bio risks, and even $100,000,000+ on asteroids (averting dinosaur-killer risk on the order of 1 in 100,000,000 per annum). At that exchange rate again a 10% x-risk impact would be worth trillions of dollars, and governments and philanthropists have shown that they are ready to spend on x-risk or GCR opportunities far, far less likely to make a counterfactual difference than 10%.

Comment author: Kawoomba 03 August 2013 08:18:44PM 1 point [-]

I see. We just used different thresholds for valuable, you used "high probability of MIRI making the counterfactual difference given survival", while for me just e.g. speeding Norvig/Gates/whoever a couple years along the path until they devote efforts to FAI would be valuable, even if it were unlikely to Make The Difference (tm).

Whoever would turn out to have solved the problem, it's unlikely that their AI safety evaluation process ("Should I do this thing?") would work in a strict vacuum, i.e. whoever will one day have evaluated the topic and made up their mind to Save The World will be highly likely to have stumbled upon MIRI's foundational work. Given that at least some of the steps in solving the problem are likely to be quite serial (sequential) in nature, the expected scenario would be that MIRI's legacy would at least provide some speed-up; a contribution which, again, I'd call valuable, even if it were unlikely to make or break the future.

If the Gates Foundation had someone evaluate the evidence for AI-related x-risk right now, you probably wouldn't expect MIRI research, AI researcher polls, philosophical essays etc. to be wholly disregarded.

Comment author: Dr_Manhattan 05 August 2013 05:14:03PM *  0 points [-]

combination of views

Sorry hard to tell from the thread which combination of views. Eliezer's?

Comment author: CarlShulman 05 August 2013 07:06:03PM 2 points [-]

The view presented by Furcas, of probable doom, and "[m]ore than 10%, definitely. Maybe 50%" probability that MIRI will be valuable given the avoidance of doom, which in the context of existential risk seems to mean averting the risk.

Comment author: Eliezer_Yudkowsky 02 August 2013 09:35:41PM 5 points [-]

...um.

It seems to me that if I believed what I infer you believe, I would be donating to MIRI while frantically trying to figure out some way to have my doomed world actually be saved.

Comment author: Furcas 02 August 2013 10:09:15PM *  3 points [-]

It seems to me that if I believed what I infer you believe, I would be donating to MIRI

Why? You (and everybody else) will almost certainly fail anyway, and you say I shouldn't multiply this low probability by the utility of saving the world.

while frantically trying to figure out some way to have my doomed world actually be saved.

The only way I see is what MIRI is doing.

Edited to add: While this is interesting, what I was really asking in my first post is, if you think the odds of MIRI succeeding are not low, why do you think so?

Comment author: Eliezer_Yudkowsky 02 August 2013 10:29:21PM 11 points [-]

Because sometimes the impossible can be done, and I don't know how to estimate the probability of that. What would you have estimated in advance, without knowing the result, was the chance of success for the AI-Box Experiment? How about if I told you that I was going to write the most popular Harry Potter fanfiction in the world and use it to recruit International Mathematical Olympiad medalists? There may be true impossibilities in this world. Eternal life may be one such, if the character of physical law is what is it appears to be, to our sorrow. I do not think that FAI is one of those. So I am going to try. We can work out what the probability of success was after we have succeeded. The chance which is gained is not gained by turning away or by despair, but by continuing to engage with and attack the problem, watching for opportunities and constantly advancing.

If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.

Comment author: Wei_Dai 05 August 2013 05:47:24AM *  9 points [-]

If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.

Multiplying small probabilities seems fine to me, whereas I really don't get "heroic epistemology".

You seem to be suggesting that "heroic epistemology" and "multiplying small probabilities" both lead to the same conclusion: support MIRI's work on FAI. But this is the case only if working on FAI has no negative consequences. In that case, "small chance of success" plus "multiplying small probabilities" warrants working on FAI, just as "medium probability of success" and "not multiplying small probabilities" does. But since working on FAI does have negative consequences, namely shortening AI timelines and (in the later stages) possibly directly causing the creation an UFAI, just allowing multiplication by small probabilities is not sufficient to warrant working on FAI if the probability of success is low.

I am really worried that you are justifying your current course of action through a novel epistemology of your own invention, which has not been widely vetted (or even widely understood). Most new ideas are wrong, and I think you ought to treat your own new ideas with deeper suspicion.

Comment author: CarlShulman 04 August 2013 04:43:40AM *  9 points [-]

heroic epistemology

Could you give a more precise statement of what this is supposed to entail?

Comment author: lukeprog 10 September 2013 09:01:49PM 1 point [-]
Comment author: Furcas 02 August 2013 11:37:09PM 0 points [-]

The most charitable way I can interpret this is:

"Yeah, the middle point of my probability interval for a happy ending is very low, but the interval is large enough that its upper bound isn't that low, so it's worth my time and your money trying to reach a happy ending."

Am I right?

feel free not to believe me about not multiplying small probabilities either.

I don't. :)

Comment author: roystgnr 02 August 2013 06:48:16PM 3 points [-]

a world in which nobody appears to be doing similar work or care sufficiently to do so.

This is astonishingly good evidence that MIRI's efforts will not be wasted via redundancy, de facto "failure" only because someone else will independently succeed first.

But it's actually (very weak) evidence against the proposition that MIRI's efforts will not be wasted because you've overestimated the problem, and it isn't evidence either way concerning the proposition that you haven't overestimated the problem but nobody will succeed at solving it.

Comment author: Eliezer_Yudkowsky 02 August 2013 08:58:48PM 2 points [-]

you're asking about the probability of having some technical people get together and solve basic research problems. I don't see why anyone else should expect to know more about that than workshop MIRI participants. Besides backward reasoning from the importance of a good result (which ordinarily operates through implying already-well-tugged ropes) is there any reason why you should be more skeptical of this than any other piece of basic research on an important problem?

Comment author: roystgnr 03 August 2013 03:40:10PM 8 points [-]

I'm concerned about the probability of having some technical people get together and solve some incredibly deep research problems before some perhaps-slightly-less-technical people plough ahead and get practical results without the benefit of that research. I'm skeptical that we'll see FAI before UFAI for the same reason I'm skeptical that we'll see a Navier-Stokes existence proof before a macroscale DNS solution, I'm skeptical that we'll prove P!=NP or even find a provably secure encryption scheme before making the world's economy dependent on unproven schemes, etc.

Even some of the important subgoals of FAI, being worked on with far more resources than MIRI has yet, are barely showing on the radar. IIRC someone only recently produced a provably correct C compiler (and in the process exposed a bunch of bugs in the industry standard compilers) - wouldn't we feel foolish if a provably FAI human-readable code turned UF simply because a bug was automatically introduced in the compilation? Or if a cosmic ray or slightly-out-of-tolerance manufacturing defect affected one of the processors? Fault-tolerant MPI is still leading-edge research, because although we've never needed it before, at exascale and above the predicted mean time between hardware-failures-on-some-node goes down to hours.

One of the reasons UFAI could be such an instant danger is the current ubiquitous nature of exploitable bugs on networked computers... yet "how do we write even simple high performance software without exploitable bugs" seems to be both a much more popular research problem than and a prerequisite to "how do we write a FAI", and it's not yet solved.

Comment author: jsteinhardt 03 August 2013 08:54:55PM 1 point [-]

I'm skeptical that we'll prove P!=NP or even find a provably secure encryption scheme before making the world's economy dependent on unproven schemes, etc.

Nitpick, but finding a provably secure encryption scheme is harder than proving P!=NP, since if P=NP then no secure encryption scheme can exist.

Comment author: Kawoomba 03 August 2013 09:20:34PM 0 points [-]

if P=NP then no [provably] secure encryption scheme can exist.

What? Why? Just because RSA would be broken? Shor's algorithm would also do so, even in a proven P!=NP world. There may be other substitutes for RSA, using different complexity classes. There are other approaches altogether. Not to mention one-time pads.

Comment author: gwern 03 August 2013 10:32:09PM 3 points [-]

As I understand it, if P=NP in a practical sense, then almost all cryptography is destroyed as P=NP destroys one-way functions & secure hashes in general. So RSA goes down, many quantum-proof systems go down, and so on and so forth, and you're left with basically just http://en.wikipedia.org/wiki/Information-theoretic_security

http://www.karlin.mff.cuni.cz/~krajicek/ri5svetu.pdf discusses some of this.

Not to mention one-time pads.

And life was so happy with just one-time pads?

Comment author: Kawoomba 03 August 2013 11:35:42PM 1 point [-]

Really, if P=NP, then encoding your messages would be quite low on the priority list ... however we're not debating the practical impact here, but that "finding a provably secure encryption scheme is harder than proving P!=NP", which was raised as a nitpick, and is clearly not the case.

Happiness or unhappiness of life with one-time pads notwithstanding.

Comment author: shminux 02 August 2013 09:48:45PM *  0 points [-]

Suppose some new rich sponsor wanted to donate a lot to MIRI, subject to an independent outside group of experts evaluating the merits of some of its core claims, like that AGI is a near-term (under 100 years) x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it. Who would you suggest s/he would engage for review?

Comment author: CarlShulman 02 August 2013 10:13:33PM *  3 points [-]

like that AGI is a near-term (under 100 years) x-risk

FHI sent a survey the top 100 most cited authors in AI and got a response rate of ~1/3, and the median estimates backed this (although this needs to be checked for response bias). Results will be published in September at PT-AI.

x-risk and that MIRI has non-negligible odds (say, a few percent or more) of mitigating it.

I.e. a probability of a few percent that there is AI risk, MIRI solves it, and otherwise it wouldn't have been solved and existential catastrophe would have resulted? That would not happen with non-gerrymandered criteria for the expert group.

But if a credible such group did deliver that result believably, then one could go to Gates or Buffett (who has spent hundreds of millions on nuclear risk efforts with much lower probability of averting nuclear war) or national governments and get billions in funding. All the work in that scenario is coming from the independent panel concluding the thing is many orders of magnitude better than almost any alternative use of spending, way past the threshold for funding.

The rich guy who says he would donate based on it is an irrelevancy in the hypo.

Comment author: Eliezer_Yudkowsky 02 August 2013 09:58:21PM 3 points [-]

Damned if I know. Oddly enough, anyone chooses to spend a bunch of their life becoming an expert on these issues tends to be sympathetic to the claims, and most random others tend to make up crap on the spot and stick with it. If they could manage to pay Peter Norvig enough money to spend a lot of time working through these issues I'd be pretty optimistic, but Peter Norvig works for Google and would be hard to pay sufficiently.

Comment author: drethelin 02 August 2013 10:55:24PM 1 point [-]

Do you guys deliberately go out of your way to evangeliz to Jaan tallin and thiel or is that source of funds a lucky break?

Comment author: lukeprog 03 August 2013 09:58:22PM *  7 points [-]

I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.

Jaan has done a surprising amount of that and also read most or all of the Sequences. Thiel has not yet decided to put in that kind of time.

Here's a list of people I'd want on that committee if they were willing to put in hundreds of hours catching up and working through the arguments with us: Scott Aaronson, Peter Norvig, Stuart Russell, Michael Nielsen.

I'd probably be able to add lots more names to that list if I could afford to spend more time becoming familiar with the epistemic standards and philosophical sophistication of more high-status CS people. I would trust Carl Shulman, Paul Christiano, Jacob Steinhardt, and a short list of others to add to my list with relatively little personal double-checking from me.

But yeah; the main problem seems to me that I don't know how to get 400 hours of Andrew Ng's time.

Although with Ng in particular it might not take 400 hours. When Louie and I met with him in Nov. '12 he seemed to think AI was almost certainly a century or more away, but by May '13 (after getting to do his deep learning work on Google's massive server clusters for a few months) he changed his tune, saying "It gives me hope –- no, more than hope –- that we might be able to [build AGI]... We clearly don’t have the right algorithms yet. It’s going to take decades. This is not going to be an easy one, but I think there’s hope.” (On the other hand, maybe he just made himself sound more optimistic than he anticipates inside because he was giving a public interview on behalf of pro-AI Google.)

Comment author: drethelin 04 August 2013 05:14:55AM 3 points [-]

This is a great answer but actually a little tangential to my question, sorry for being vague. Mine was actually about the part of shminux's proposal that involved finding potential mega donors. Relatedly, how much convincing do you think it would take to get Tallinn or thiel to increase their donations by an order of magnitude, something they could easily afford? This seems like a relatively high leverage plan if you can swing it. With x million dollars you can afford to actually pay to hire people like google can, if on a much smaller scale.

Comment author: wedrifid 02 August 2013 01:56:38PM 1 point [-]

Success is defined most obviously as actually constructing an FAI, but of course if our work were picked up elsewhere and reused after MIRI itself died as an organization for whatever reason,

(Or if it were picked up elsewhere and MIRI is merely overtaken. Dying isn't necessary.)

Comment author: ColonelMustard 07 October 2013 02:32:06AM 4 points [-]

I am thinking of writing a discussion thread to propose MIRI make it a priority to create a (video/pamphlet/blog post), tailored to intelligent non-rationalists and with as little jargon as possible (e.g. no terms like Kolmogorov complexity), to explain the dangers of UFAI. Please upvote this comment if you think LW is better with such a post, because I have zero karma.

Comment author: shminux 01 August 2013 10:00:46PM -1 points [-]

How does MIRI Know it Has a Medium Probability of Success?

it doesn't know of course. The uncertainty in any probability estimate is bound to be huge, which is reflected in the variety of estimates and opinions event within MIRI. I hope to see some day a thoughtfully constructed reference class with a large enough number of members to be worth drawing conclusions from.

Comment author: Rain 01 August 2013 02:05:50PM *  1 point [-]

I give them about 10 percent probability, and statements I've read from Eliezer and Luke cause me to believe they're below medium in self-assessment nowadays, or their definition of "medium" may be lower than you think. I support them to increase the chance.

That said, low chances of success aren't good for recruiting or motivation, and maximizing the probability requires as much effort and skill as possible to obtain.

Comment author: peter_hurford 01 August 2013 03:11:15PM 6 points [-]

1.) What do you base the 10% estimate on?

2.) Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument. What do you think?

Comment author: Rain 01 August 2013 03:19:47PM *  7 points [-]

1.) What do you base the 10% estimate on?

I base it on everything I've read and seen on technology, human nature, historical uses of power, trends in tech capabilities, the effects of intelligence, MIRI's mission, team, and focus, and the greater realm of philanthropic endeavors.

If you mean, 'you pulled that number out of your butt, and therefore I call you on it,' then I'll have to admit defeat due to inability to break it down quantitatively. Sorry.

2.) Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument. What do you think?

I think that's taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a 'lever point' which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.

Comment author: peter_hurford 01 August 2013 06:29:28PM *  2 points [-]

If you mean, 'you pulled that number out of your butt, and therefore I call you on it,' then I'll have to admit defeat due to inability to break it down quantitatively. Sorry.

Yeah. On one hand, I think there is something to be said about needing to make these fast and loose estimates and that there's some basis for them. But on the other hand, I think one needs to recognize just how fast and loose they are. I think our error bars on MIRI's chance of success is really high.

~

I think that's taken out of context. The way I understand it, he means superintelligence will have a really high impact regardless (near 100% probability), and is therefore a 'lever point' which can have a higher probability of being impacted by anyone paying attention to it, and since MIRI is one of very few groups paying attention, they have a medium probability of being such an impactor.

Let me put that in premise-conclusion form:

P1: Superintelligence will, with probability greater than 99.999%, dramatically impact the future.

P2: One can change how superintelligence will unfold by working on superintelligence.

C3: Therefore from P1 and P2, working on superintelligence will dramatically impact the future.

P4: MIRI is one of the only groups working on superintelligence.

C5: Therefore from C3 and P4, MIRI will dramatically impact the future.

Do you think that's right?

If so, I think P2 could be false, but I'll accept it for the sake of argument. The real problem is, I think, C5 is a fallacy. It either assumes that any work in the domain will affect how superintelligence unfolds in a controlled way (which seems false) or that MIRI's work will have impact (which seems unproven).

Comment author: Nornagest 01 August 2013 09:16:45PM *  6 points [-]

P1 is almost certainly an overestimate: independent of everything else, there's almost certainly a greater than 0.001% chance that a civilization-ending event will occur before anyone gets around to building a superintelligence. The potential importance of AI research by way of this chain of logic wouldn't be lowered too much if you used 80 or 90%, though.

Comment author: Rain 01 August 2013 06:58:22PM *  0 points [-]

I'm not sure which fallacy you're invoking, but saying (to paraphrase), 'superintelligence is likely difficult to aim' and 'MIRI's work may not have an impact' are certainly possible, and already contribute to my estimates.

Comment author: peter_hurford 02 August 2013 07:28:51AM 0 points [-]

I think a fair amount of people argue that because a cause is important, anyone working on that cause must be doing important work.

Comment author: Rain 02 August 2013 03:21:26PM 2 points [-]

The method is even more important (practice vs. perfect practice, philanthropy vs. givewell). I believe in the mission, not MIRI per se. If Eliezer decided that magic was the best way to achieve FAI and started searching for the right wand and hand gestures rather than math and decision theory, I would look elsewhere.

Comment author: Larks 02 August 2013 08:52:06AM 9 points [-]

Eliezer is very much against the idea of supporting MIRI based on a "low probability of really high impact" argument.

I hate to put words in his mouth, but I think

  • he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).
  • Eliezer thinks the case for MIRI does not depend on very small chances, and furthermore, is sceptical that these arguments are used in practice by Xrisk organisations, etc. He wouldn't necessary turn away someone's money who said "I'm donating because of a 10^-10 chance." (though equally he might for PR/paternalistic reasons)
Comment author: peter_hurford 02 August 2013 07:39:05PM 0 points [-]

he means 0.0001% chance, not 10% chance. 10% is well within the range of probabilities humans can reason about (to the extent that humans can reason about any probabilities).

Where does this 10% probability come from?

Comment author: Rain 02 August 2013 10:56:21PM *  2 points [-]

Anchoring from my butt-number?

Comment author: lukeprog 03 August 2013 10:02:34PM 1 point [-]

I believe the correct term is "ass-pull number." :)

Comment author: gjm 01 August 2013 04:28:11PM 1 point [-]

Could you clarify your definition of success?

Your PredictionBook link says you reckon 10% probability of humanity still being around in ~100 years, but that's not the same thing as MIRI succeeding. Superhuman AI might turn out to be beyond human capability, so we could survive without MIRI achieving anything. Superhuman UFAI might be feasible, and MIRI might successfully stop it happening (in which case I'd say they succeeded), but FAI might be just too hard or too weak and we might then get wiped out by something else. (I agree that that seems low probability.)

Comment author: Rain 01 August 2013 04:30:57PM *  1 point [-]

Could you clarify your definition of success?

From MIRI's mission statement: "the creation of smarter-than-human intelligence has a positive impact."

I see smarter-than-human intelligence as required to overcome the combined threat of existential risks in the long run.

Comment author: Pablo_Stafforini 02 August 2013 01:37:28AM *  4 points [-]

The full sentence reads: "MIRI exists to ensure that the creation of smarter-than-human intelligence has a positive impact." (emphasis added) Clearly, if smarter-than-human intelligence ends up having a positive impact independently (or in spite of) MIRI's efforts, that would count as a success only in a Pickwickian sort of sense. To succeed in the sense obviously intended by the authors of the mission statement, MIRI would have to be at least partially causally implicated in the process leading to the creation of FAI.

So the question remains: on what grounds do you believe that, if smarter-than-human intelligence ends up having a positive impact, this will be necessarily at least partly due to MIRI's efforts? I find that view implausible, and instead agree with Carl Shulman that "the impact of MIRI in particular has to be far smaller subset of the expected impact of the cause as a whole," for the reasons he mentions.

Comment author: Rain 02 August 2013 02:05:33PM *  2 points [-]

I subscribe to the view that AGI is bad by default, and don't see anyone else working on the friendliness problem.

Comment author: Omid 01 August 2013 04:21:55PM 1 point [-]

So we're probably doomed then?

Comment author: CarlShulman 01 August 2013 07:21:01PM *  3 points [-]

Surveys at the FHI's Winter Intelligence conference, and sent out to AI experts showed median estimates of existential risk from AI as less likely than good outcomes from AI.

I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.

Comment author: JonahSinick 01 August 2013 10:08:35PM 1 point [-]

I would say we are probably not doomed by that channel, as would, e.g. Paul Christiano, although I would say (as do the surveys, including of neutral AI experts) the risk is significant.

I agree with this, but I would be interested in knowing your own reasons for saying that we're probably not doomed via AI risk. I spelled out some reasons for my own belief here.

Comment author: Yosarian2 02 August 2013 10:31:04PM 0 points [-]

Personally, I don't think that there is that high of a chance that we're doomed via AI risk because I think that the odds of "AI goes foom" are significantly lower then the MIRI people do. I think it's somewhat more likely that subhuman-level GAI's and vaugly human level AI's increase the rate of AI research somewhat, but not to the dramatic levels the EY thinks. With a slower paced rate of AI development, we're probably not dealing with a singleton AI, we have time to work to perfect friendly AI research while getting assistance from roughly human level AI's, while slowly developing the slightly smarter AI's, and while improving friendly-AI software with every step.

I don't think the "AI goes foom" scenario is impossible, but I don't put it at a high likelihood, maybe 10%-20%. I just don't think that it's all that likely that a human level GAI (say, one with the equivilent of IQ-90 to IQ-100) is able to rapidly turn itself into a IQ-300 GAI; if we can't do it that quickly, then a AI of similar intelligence shouldn't be able to do much better. And in the slower AI takeoff scenario I think is more likely, there is still some AI risk, but research in friendly AI we do now at this early stage is likely to be mostly made obsolete by research we do at that stage, and we're more likely to adapt and tweak solutions rather then having to have a totally flawless solution ready the day before someone develops the first moderately useful-level GAI.

So i would probably put the total odds of MIRI directly having a big impact to be maybe around 5%. Which, considering the stakes involved, is still very significant and worth the cost, and there is also a significant change that their work has other useful spinoffs even if it turns out to not be necessary for the reason intended.

Comment author: Rain 01 August 2013 04:31:44PM *  0 points [-]

I'm pessimistic and depressed.