If this was a regular math problem and it wasn't world-shakingly important, why wouldn't you expect that funding workshops and then researchers would cause progress on it?
Assigning a very low probability to progress rests on a sort of backwards reasoning wherein you expect it to be difficult to do things because they are important. The universe contains no such rule. They're just things.
It's hard to add a significant marginal fractional pull to a rope that many other people are pulling on. But this is not a well-tugged rope!
I'm not assigning a low probability to progress, I'm assigning a low probability to success.
Where FAI research is concerned, progress is only relevant in as much as it increases the probability of success, right?
Unlike a regular math problem, you've only got one shot at getting it right, and you're in a race with other researchers who are working on an easier problem (seed AI, Friendly or not). It doesn't matter if you're 80% of the way there if we all die first.
Edited to add and clarify: Even accounting for the progress I think you're likely to make, the probability of success remains low, and that's what I care about.
In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?
I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.
-
(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)