In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?
I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.
-
(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)
Yeah. On one hand, I think there is something to be said about needing to make these fast and loose estimates and that there's some basis for them. But on the other hand, I think one needs to recognize just how fast and loose they are. I think our error bars on MIRI's chance of success is really high.
~
Let me put that in premise-conclusion form:
P1: Superintelligence will, with probability greater than 99.999%, dramatically impact the future.
P2: One can change how superintelligence will unfold by working on superintelligence.
C3: Therefore from P1 and P2, working on superintelligence will dramatically impact the future.
P4: MIRI is one of the only groups working on superintelligence.
C5: Therefore from C3 and P4, MIRI will dramatically impact the future.
Do you think that's right?
If so, I think P2 could be false, but I'll accept it for the sake of argument. The real problem is, I think, C5 is a fallacy. It either assumes that any work in the domain will affect how superintelligence unfolds in a controlled way (which seems false) or that MIRI's work will have impact (which seems unproven).
P1 is almost certainly an overestimate: independent of everything else, there's almost certainly a greater than 0.001% chance that a civilization-ending event will occur before anyone gets around to building a superintelligence. The potential importance of AI research by way of this chain of logic wouldn't be lowered too much if you used 80 or 90%, though.