As it turns out, it is perfectly rational to inspect evidence that contradicts your beliefs more closely than evidence that confirms them. If evidence goes against your beliefs it is more likely to be fake evidence. As the saying goes, your strength as a rationalist is your ability to detect fabricated evidence.
EDIT: As Ben Elliot points out in this post this argument really only applies if you happen to be a perfect Bayesian (and you aren't). In real life you're biased toward confirmatory evidence, and so often you really should check it over disconfirmatory evidence. However, it is worth keeping track of which things you're doing to compensate for a human bias, and which things you're doing because they're optimal for Bayesians.
No. This is an easy mistake to make, but it has bad consequences. Detecting fake evidence is only an instrumental goal, in service to the more important goal of maximizing the accuracy of your beliefs. There is a common trap, where all the evidence you see, on both sides, would fall if you challenged it, but you only challenge the pieces you disagree with; and then you add up the remaining, also-invalid evidence, and conclude that you were right all along. It is theoretically possible to counter this bias directly, by discounting confirmatory evidence that...
This is our monthly thread for collecting these little gems and pearls of wisdom, rationality-related quotes you've seen recently, or had stored in your quotesfile for ages, and which might be handy to link to in one of our discussions.