One of the most useful techniques of rationality is taking the outside view, also known as reference class forecasting. Instead of thinking too hard about particulars of a given situation and taking a guess which will invariably turned out to be highly biased, one looks at outcomes of situations which are similar in some essential way.
Figuring out correct reference class might sometimes be difficult, but even then it's far more reliable than trying to guess while ignoring the evidence of similar cases. Now in some situations we have precise enough data that inside view might give correct answer - but for almost all such cases I'd expect outside view to be as usable and not far away in correctness.
Something that keeps puzzling me is persistence of certain beliefs on lesswrong. Like belief in effectiveness of cryonics - reference class of things promising eternal (or very long) life is huge and has consistent 0% success rate. Reference class of predictions based on technology which isn't even remotely here has perhaps non-zero but still ridiculously tiny success rate. I cannot think of any reference class in which cryonics does well. Likewise belief in singularity - reference class of beliefs in coming of a new world, be it good or evil, is huge and with consistent 0% success rate. Reference class of beliefs in almost omnipotent good or evil beings has consistent 0% success rate.
And many fellow rationalists not only believe that chances of cryonics or singularity or AI are far from negligible levels indicated by the outside view, they consider them highly likely or even nearly certain!
There are a few ways how this situation can be resolved:
- Biting the outside view bullet like me, and assigning very low probability to them.
- Finding a convincing reference class in which cryonics, singularity, superhuman AI etc. are highly probable - I invite you to try in comments, but I doubt this will lead anywhere.
- Or is there a class of situations for which the outside view is consistently and spectacularly wrong; data is not good enough for precise predictions; and yet we somehow think we can predict them reliably?
How do you reconcile them?
The human brain is made out of matter (materialism). Many people's brains are largely intact at the time of their deaths. By preserving the brain, we give possible future advanced neuroscience and materials technology a chance at restoring the original person. There are certainly a number of good reasons to think that this probably won't happen, but it doesn't belong in the same reference class of "predictions promising eternal life," because most previous predictions about about eternal life didn't propose technological means in a material universe. Cryonics isn't about rapturing people's souls up to heaven; it's about reconstruction of a damaged physical artifact. Conditional on continued scientific progress (which might or might not happen), it seems plausible. I do agree that "technology which isn't even remotely here" is a good reference class. Similarly ...
Intelligence doesn't require ontologically fundamental things that we can't create more of, only matter appropriately arranged (materialism). Humans are not the most powerful possible intelligences (mediocrity). Conditional on continued scientific progress, it's plausible that we could create superhuman AIs.
Human minds are not the fastest-thinking or the fastest-improving possible intelligences (mediocrity). Faster processes outrun slower ones. Conditional on our creating AIs, some of them might think much faster than us, and faster minds probably have a greater share in determining the future.
These are fine arguments, but they all take the inside view - focusing on particulars of a situation, not finding big robust reference classes to which the situation belongs.
And in any case you seem to be arguing for such inventions not being prohibited by laws of physics more than for them happening with very high probability in near future, as many here believe. As a reference class, things which are merely not prohibited by laws of physics almost never happen anyway - this class is just too huge.