Eliezer, overall a very good post. As usual, I'm somewhat maddened by lower quality thought/writing mixed in with your very best thought/writing.

In particular, your claim that competent scientists make detailed predictions about 2050 because they're unaware of the conjunction fallacy or representativeness heuristics fits in a long term trend by OB bloggers (and affiliated bloggers) that annoys me: you pretend that any performed belief is an actual belief. Whether it is or not is an empirical question. But in a way you're rather ruthlessly siding with predicting scientist themselves, against those of us who would rather look at phenomena like that more critically, by taking them at their word that their expressed belief is their actual belief.

I like that you're dropping the science vs. bayescraft line here, and focusing more on weaknesses in the search for knowledge and understanding as performed by science, and how it can be improved by insights from bayesian probability/reasoning.