In the beginning came the idea that we can't just toss out Aristotle's armchair reasoning and replace it with different armchair reasoning. We need to talk to Nature, and actually listen to what It says in reply. This, itself, was a stroke of genius.
If you do a probability-theoretic calculation correctly, you're going to get the rational answer.
How does one make sure that this "probability-theoretic calculation" is not a "different armchair reasoning"?
Science doesn't trust your rationality, and it doesn't rely on your ability to use probability theory as the arbiter of truth. It wants you to set up a definitive experiment. [...] Science is built around the assumption that you're too stupid and self-deceiving to just use Solomonoff induction.
This seems like a safe assumption. On the other hand, trusting in your powers of Solomonoff induction and Bayesianism doesn't seem like one: what if you suck at estimating priors and too unimaginative to account for all the likely alternatives?
So, are you going to believe in faster-than-light quantum "collapse" fairies after all? Or do you think you're smarter than that?
Again a straw-collapse. No one believes in faster-than-light quantum "collapse", except for maybe some philosophers of physics.
... Solomonoff induction ...
Totally agreed. Thing is in general incomputable, how much more you need not to trust yourself doing it correctly? Clearly you can't have a process that relies on computing incomputable things right. I'm becoming increasingly convinced, either via confirmation bias, or via proper updates, that Eliezer skipped helluva lot of fundamentals.
Today's post, Science Doesn't Trust Your Rationality was originally published on 14 May 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The Dilemma: Science or Bayes?, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.