The basic questions of rationality

I've been on Less Wrong since its inception, around March 2009. I've read a lot and contributed a lot, and so now I'm more familiar with our jargon, I know of a few more scientific studies, and I might know a couple of useful tricks. Despite all my reading, however, I feel like I'm a far cry from learning rationality. I'm still a wannabe, not an amateur. Less Wrong has tons of information, but I feel like I haven't yet learned the answers to the basic questions of rationality.

I, personally, am a fan of the top-down approach to learning things. Whereas Less Wrong contains tons of useful facts that could, potentially, be put together to answer life's important questions, I really would find it easier if we started with the important questions, and then broke those down into smaller pieces that can be answered more easily.

And so, that's precisely what I'm going to do. Here are, as far as I can tell, the basic questions of rationality—the questions we're actually trying to answer here—along with what answers I've found:

Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.

Q: How can we effectively gather relevant evidence? A: I don't know. (Controlled experiments? Asking people?)
Q: How can we correctly apply reason? A: If you have infinite computational resources available, use probability theory.
- Q: We don't have infinite computational resources available, so what now? A: I don't know. (Apply Bayes' rule anyway? Just try to emulate what a hypercomputer would do?)
Q: How can we successfully apply intuition? A: By repairing our biases, and developing habits that point us in the right direction under specific circumstances.
- Q: How can we find our biases? A: I don't know. (Read Less Wrong? What about our personal quirks? How can we notice those?)
- Q: Once we find a bias, how can we fix it? A: I don't know. (Apply a correction, test, repeat? Figure out how the bias feels?)
- Q: How can we find out what habits would be useful to develop? A: I don't know. (Examine our past successes and rationalize them?)
- Q: Once we decide on a habit, how can we develop it? A: I don't know. (Sheer practice?)

We could answer some of these questions ourselves, though simple practice and straightforward methods. The method "apply a correction, test, repeat", for example, is so generally useful that it deserves to be called the Fundamental Algorithm of Control. Nevertheless, since Less Wrong is devoted to developing human rationality, surely it contains answers to these questions somewhere. Where are they?

Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.

Q: How can we effectively gather relevant evidence? A: I don't know. (Controlled experiments? Asking people?)
Q: How can we correctly apply reason? A: If you have infinite computational resources available, use probability theory.
- Q: We don't have infinite computational resources available, so what now? A: I don't know. (Apply Bayes' rule anyway? Just try to emulate what a hypercomputer would do?)
Q: How can we successfully apply intuition? A: By repairing our biases, and developing habits that point us in the right direction under specific circumstances.
- Q: How can we find our biases? A: I don't know. (Read Less Wrong? What about our personal quirks? How can we notice those?)
- Q: Once we find a bias, how can we fix it? A: I don't know. (Apply a correction, test, repeat? Figure out how the bias feels?)
- Q: How can we find out what habits would be useful to develop? A: I don't know. (Examine our past successes and rationalize them?)
- Q: Once we decide on a habit, how can we develop it? A: I don't know. (Sheer practice?)

If we had a formal and efficient answer to this we would have FAI by now* or determined it impossible!

*Answering the question "Given a question, how should we go about answering it?" seems AI complete if not super human AI complete. People answer questions without doing it the right bayesian bless-ed way all the time with a bit of luck and brute force.

Answering the question "Given a question, how should we go about answering it?" seems AI complete if not super human AI complete.

Well, yes, that's kind of the point. Rationality is about finding the answers to all useful questions; we can achieve this by finding good answers to fully general questions, like that one.

40

The basic questions of rationality

40

40

40

The basic questions of rationality

40

40