Apply Bayes' rule anyway. The result will not be perfect and you should be aware of that, but in majority of situations it's still an improvement over intuitive guesses.
How do you determine the relevant probabilities? What if you're looking for, say, the probability of a nuclear attack occurring anywhere in the world in the next 20 years?
Q: Once we find a bias, how can we fix it?
Retreat to more formalised reasoning, if possible.
Yes, but that doesn't remove the bias. Surely if it's at all possible to remove a bias, that's better than circumventing it through formal reasoning, because formal reasoning is much slower than intuition.
How do you determine the relevant probabilities? What if you're looking for, say, the probability of a nuclear attack occurring anywhere in the world in the next 20 years?
What information you are updating from?
I've been on Less Wrong since its inception, around March 2009. I've read a lot and contributed a lot, and so now I'm more familiar with our jargon, I know of a few more scientific studies, and I might know a couple of useful tricks. Despite all my reading, however, I feel like I'm a far cry from learning rationality. I'm still a wannabe, not an amateur. Less Wrong has tons of information, but I feel like I haven't yet learned the answers to the basic questions of rationality.
I, personally, am a fan of the top-down approach to learning things. Whereas Less Wrong contains tons of useful facts that could, potentially, be put together to answer life's important questions, I really would find it easier if we started with the important questions, and then broke those down into smaller pieces that can be answered more easily.
And so, that's precisely what I'm going to do. Here are, as far as I can tell, the basic questions of rationality—the questions we're actually trying to answer here—along with what answers I've found:
Q: Given a question, how should we go about answering it? A: By gathering evidence effectively, and correctly applying reason and intuition.