The basic questions of rationality

I've been on Less Wrong since its inception, around March 2009. I've read a lot and contributed a lot, and so now I'm more familiar with our jargon, I know of a few more scientific studies, and I might know a couple of useful tricks. Despite all my reading, however, I feel like I'm a far cry from learning rationality. I'm still a wannabe, not an amateur. Less Wrong has tons of information, but I feel like I haven't yet learned the answers to the basic questions of rationality.

I, personally, am a fan of the top-down approach to learning things. Whereas Less Wrong contains tons of useful facts that could, potentially, be put together to answer life's important questions, I really would find it easier if we started with the important questions, and then broke those down into smaller pieces that can be answered more easily.

And so, that's precisely what I'm going to do. Here are, as far as I can tell, the basic questions of rationality—the questions we're actually trying to answer here—along with what answers I've found:

Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.

Q: How can we effectively gather relevant evidence? A: I don't know. (Controlled experiments? Asking people?)
Q: How can we correctly apply reason? A: If you have infinite computational resources available, use probability theory.
- Q: We don't have infinite computational resources available, so what now? A: I don't know. (Apply Bayes' rule anyway? Just try to emulate what a hypercomputer would do?)
Q: How can we successfully apply intuition? A: By repairing our biases, and developing habits that point us in the right direction under specific circumstances.
- Q: How can we find our biases? A: I don't know. (Read Less Wrong? What about our personal quirks? How can we notice those?)
- Q: Once we find a bias, how can we fix it? A: I don't know. (Apply a correction, test, repeat? Figure out how the bias feels?)
- Q: How can we find out what habits would be useful to develop? A: I don't know. (Examine our past successes and rationalize them?)
- Q: Once we decide on a habit, how can we develop it? A: I don't know. (Sheer practice?)

We could answer some of these questions ourselves, though simple practice and straightforward methods. The method "apply a correction, test, repeat", for example, is so generally useful that it deserves to be called the Fundamental Algorithm of Control. Nevertheless, since Less Wrong is devoted to developing human rationality, surely it contains answers to these questions somewhere. Where are they?

Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.

Q: How can we effectively gather relevant evidence? A: I don't know. (Controlled experiments? Asking people?)
Q: How can we correctly apply reason? A: If you have infinite computational resources available, use probability theory.
- Q: We don't have infinite computational resources available, so what now? A: I don't know. (Apply Bayes' rule anyway? Just try to emulate what a hypercomputer would do?)
Q: How can we successfully apply intuition? A: By repairing our biases, and developing habits that point us in the right direction under specific circumstances.
- Q: How can we find our biases? A: I don't know. (Read Less Wrong? What about our personal quirks? How can we notice those?)
- Q: Once we find a bias, how can we fix it? A: I don't know. (Apply a correction, test, repeat? Figure out how the bias feels?)
- Q: How can we find out what habits would be useful to develop? A: I don't know. (Examine our past successes and rationalize them?)
- Q: Once we decide on a habit, how can we develop it? A: I don't know. (Sheer practice?)

It's possible that one can learn the wrong kind of rationality first, but I disagree with the idea that rationality can be a bad thing in general.

The first skill ought to be efficient usage of computational resources. For humans, that means calories (no longer a factor outside our ancestral environment) and time. LessWrong has taught me exceptional methods of distinguishing my exact preferences and mapping out every contour of my own utility function. Thus, I can decide exactly which flavor Ice Cream will give me more utilons. This is neat, but useless.

But rationality is more than just thinking. It's thinking about thinking. And thinking about thinking about thinking. And it keeps recursing. In learning the difference in utility between Ice Cream, I learned to never care about which flavor I will get; the utility saved by not consciously calculating it is worth more than the utility of thinking about it. Before learning about rationality, I had been the sort to whinge and waver over flavors at coldstone. After learning some rationality, I was even worse. But learning a bit more of rationality, well learning the exact right bit that is, made me perform at peak speed and peak utility. It's a lesson I've learned that carries over into more than just food, has saved me countless hours and utilons, and will probably stay with me until the end of my days.

There are many valleys of bad rationality. Whether they take the form of being just clever enough to choose the 'rational' position in game theory versus the 'super-rational', or just smart enough to catch a mimetic autoimmune disease without fully deconverting from your religion, rationality valleys can be formidable traps. As always, though the valley may be long and hard, the far side of the valley is always better than where you started.

In short: a method of answering questions should be judged not only on its benefits, but on its costs. So, another basic question of rationality is:

Q: When should we stop thinking about a question?

40

The basic questions of rationality

40

40

40

The basic questions of rationality

40

40