In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?
I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.
-
(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)
No they don't; they could be checking relative plausibility of causing an OK outcome without trying to put absolute numbers on a probability estimate, and this is reasonable due to the following circumstances:
The life lesson I've learned is that by the time you really get anywhere, if you get anywhere, you'll have encountered some positive miracles, some negative miracles, your plans will have changed, you'll have found that the parts which took the longest weren't what you thought they would be, and that other things proved to be much easier than expected. Your successes won't come through foreseen avenues, and neither will your failures. But running through it all will be the fundamental realization that everything you accomplished, and all the unforeseen opportunities you took advantage of, were things that you would never have received if you hadn't attacked the hardest part of the problem that you knew about straight-on, without distraction.
How do you estimate probabilities like that? I honestly haven't a clue. Now, we all still have to maximize expected utility, but the heuristic I'm applying to do that (which at the meta level I think is the planning heuristic with the best chance of actually working) is to ask "Is there any better way of attacking the hardest part of the problem?" or "Is there any better course of action which doesn't rely on someone else performing a miracle?" So far as I can tell, these other proposed courses of action don't attack the hardest part of the problem for humanity's survival, but rely on someone else performing a miracle. I cannot make myself believe that this would really actually work. (And System 2 agrees that System 1's inability to really believe seems well-founded.)
Since I'm acting on such reasons and heuristics as "If you don't attack the hardest part of the problem, no one else will" and "Beware of taking the easy way out" and "Don't rely on someone else to perform a miracle", I am indeed willing to term what I'm doing "heroic epistemology". It's just that I think such reasoning is, you know, actually correct and normative under these conditions.
If you don't mind mixing the meta-level and the object-level, then I find any reasoning along the lines of "The probability of our contributing to solving FAI is too low, maybe we can have a larger impact by working on synthetic biology defense and hoping a miracle happens elsewhere" much less convincing than the meta-level observation, "That's a complete Hail Mary pass, if there's something you think is going to wipe out humanity then just work on that directly as your highest priority." All the side cleverness, on my view, just adds up to losing the chance that you get by engaging directly with the problem and everything unforeseen that happens from there.
Another way of phrasing this is that if we actually win, I fully expect the counterfactual still-arguing-about-this version of 2013-Carl to say, "But we succeeded through avenue X, while you were then advocating avenue Y, which I was right to say wouldn't work." And to this the counterfactual reply of Eliezer will be, "But Carl, if I'd taken your advice back then, I wouldn't have stayed engaged with the problem long enough to discover and comprehend avenue X and seize that opportunity, and this part of our later conversation was totally foreseeable in advance." Hypothetical oblivious!Carl then replies, "But the foreseeable probability should still have been very low" or "Maybe you or someone else would've tried Y without that detour, if you'd worked on Z earlier" where Z was not actually uniquely suggested as the single best alternative course of action at the time. If there's a reply that counterfactual non-oblivious Carl can make, I can't foresee it from here, under those hypothetical circumstances unfolding as I describe (and you shouldn't really be trying to justify yourself under those hypothetical circumstances, any more than I should be making excuses in advance for what counterfactual Eliezer says after failing, besides "Oops").
My reasoning here is, from my internal perspective, very crude, because I'm not sure I really actually trust non-crude reasoning. There's this killer problem that's going to make all that other stuff pointless. I see a way to make progress on it, on the object level; the next problem up is visible and can be attacked. (Even this wasn't always true, and I stuck with the problem anyway long enough to get to the point where I could state the tiling problem.) Resources should go to attacking this visible next step on the hardest problem. An exception to this as top priority maximization was CFAR, via "teaching rationality demonstrably channels more resources toward FAI; and CFAR which will later be self-sustaining is just starting up; plus CFAR might be useful for a general saving throw bonus; plus if a rational EA community had existed in 1996 it would have shaved ten years off the timeline and we could easily run into that situation again; plus I'm not sure MIRI will survive without CFAR". Generalizing, young but hopefully self-sustaining initiatives can be plausibly competitive with MIRI for small numbers of marginal dollars, provided that they're sufficiently directly linked to FAI down the road. Short of that, it doesn't really make sense to ignore the big killer problem and hope somebody else handles it later. Not really actually.
Also, because this.