The point is that all object-level arguments for and against these scenarios, even if you call them "probability estimates", are ultimately based on intuitions which are difficult to formalize or quantify.
The scenarios hypthesized by the Singularitarians are extreme, both in the magnitude of the effect they are claimed to entail, and in the the highly conjunctive object-level arguments that are used to argue for them. Common sense rationality tells us that "extraordinary claims demand exceptional evidence". How do we evaluate whether the intuitions of these people constitute "exceptional evidence"?
So we take the "outside view" and try to meta-reason on these arguments and the people making them:
Can we trust their informal intuitions, or do they show any signs of classical biases?
Are these people privileging the hypothesis? Are they drawing their intuitions from the availability heuristic?
If intelligence explosion/cryonics/all things singularitarian were ideas radically different from any common meme, then the answer to these questions would be likely no: these ideas would appear counterintuitive at a gut level to most normally rational people, possibly in the same way quantum mechanics and Einstenian relativity appear conterituitive.
If domain-level experts, after studing the field for years, recalibrated their intuitions and claimed that these scenarios were likely, then we should probably listen to them.
We should not just accept their claims based on authority, of course: even the experts can subject to groupthink and other biases (cough...economists...cough), but as far as the "outside view" is concerned, we would at least have plausibly excluded the availability bias.
What we observe, instead, is that singulariarians ideas strongly pattern-match to Christian millenarianism and similar religious beliefs, mixed with popular scifi tropes (cryonics, AI revolt, etc.). They certainly originated, or at least were strongly influenced by these memes, and therefore the intuitions of the people arguing for them are likely "contaminated" via the availability heuristic by these memes.
More specifically, if singulariarians ideas make intuitive sense to you, you can't even trust your own intuitions since they are likely to be "contaminated" as well.
Add the fact that the strength of these intuitions seems to decrease rather than increase with domain-expertise, suggesting that the Dunning–Kruger effect is also at work, then the "outside view" tells us to be wary.
Of course, it is possible to believe correct things even when they are likely to be the subject of biases, or even to believe correct things that many people believe for the wrong reason, but in order to make a case for these beliefs, you need some airtight arguments with strong evidence.
As far as I can tell, MIRI/FHI/other Singularitarians have provided no such arguments.
Between outside view, Dunning-Krueger, and rhetorical questions about biases with no attempt to provide answers to them, you've built a schema for arguing against anything at all without the burden of bringing evidence to the table. I guess evidence would be the dreaded inside view, although that doesn't stop you demanding it from the other side. Bostrom's recent book? The arguments in the Sequences? No, that doesn't count, it's not exceptional enough, and besides, Dunning-Krueger means no-one ever knows they're wrong, and (contd. p.94).
Maybe a better name for "outside view" would be "spectator's view", or "armchair view".
I'm giving a talk to the Boulder Future Salon in Boulder, Colorado in a few weeks on the Intelligence Explosion hypothesis. I've given it once before in Korea but I think the crowd I'm addressing will be more savvy than the last one (many of them have met Eliezer personally). It could end up being important, so I was wondering if anyone considers themselves especially capable of playing Devil's Advocate so I could shape up a bit before my talk? I'd like there to be no real surprises.
I'd be up for just messaging back and forth or skyping, whatever is convenient.