How does MIRI Know it Has a Medium Probability of Success?

Peter Wildeford

In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?

I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.

(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)

In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?

I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.

(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)

Because sometimes the impossible can be done, and I don't know how to estimate the probability of that. What would you have estimated in advance, without knowing the result, was the chance of success for the AI-Box Experiment? How about if I told you that I was going to write the most popular Harry Potter fanfiction in the world and use it to recruit International Mathematical Olympiad medalists? There may be true impossibilities in this world. Eternal life may be one such, if the character of physical law is what is it appears to be, to our sorrow. I do not think that FAI is one of those. So I am going to try. We can work out what the probability of success was after we have succeeded. The chance which is gained is not gained by turning away or by despair, but by continuing to engage with and attack the problem, watching for opportunities and constantly advancing.

If you don't believe me about that aspect of heroic epistemology, feel free not to believe me about not multiplying small probabilities either.

13Wei Dai13y

Multiplying small probabilities seems fine to me, whereas I really don't get "heroic epistemology". You seem to be suggesting that "heroic epistemology" and "multiplying small probabilities" both lead to the same conclusion: support MIRI's work on FAI. But this is the case only if working on FAI has no negative consequences. In that case, "small chance of success" plus "multiplying small probabilities" warrants working on FAI, just as "medium probability of success" and "not multiplying small probabilities" does. But since working on FAI does have negative consequences, namely shortening AI timelines and (in the later stages) possibly directly causing the creation an UFAI, just allowing multiplication by small probabilities is not sufficient to warrant working on FAI if the probability of success is low. I am really worried that you are justifying your current course of action through a novel epistemology of your own invention, which has not been widely vetted (or even widely understood). Most new ideas are wrong, and I think you ought to treat your own new ideas with deeper suspicion.

17CarlShulman13y

Could you give a more precise statement of what this is supposed to entail?

35

How does MIRI Know it Has a Medium Probability of Success?

35

35

35

How does MIRI Know it Has a Medium Probability of Success?

35

35