The General Contrarian Heuristic:
Assume these and such people who claim to be right actually are at-least-somewhat-straightforwardly right, and they have good evidence or arguments that you're just not aware of. (There are many plausible reasons for your ignorance; e.g. for the longest time I thought Christianity and ufology were just obviously stupid, because I'd only read atheist/skeptic/scientismist diatribes. What evidence filtered evidence?) What is the most plausible evidence or argument that can be found while searching in good faith? This often splits in two directions:
Some contrarian topics I've had fun exploring:
Assume UFO phenomena and Marian apparitions are legit, i.e. caused by some transhumanly powerful process. E.g., the Miracle at Fatima. What would be the mechanism? More pertinently, what would be the motivations?
Assume legit retroscausal psi effects in parapsychology: What would be the mechanism?
Assuming it is legit, i.e. retrocausal results are legit, why is psi capricious?
Assume intelligent life isn't fantastically unlikely. Why no signs of intelligent life? (Related to "why is psi capricious" question.)
Remember, skepticism is easy, it's the default position: if the phenomenon you're modeling is actually complex, your explanation will have to be subtle. It's always too easy to shout "confirmation bias", "mass hallucination", "memetic selection pressures", and what have you. Don't fall for that trap; it's just as much of an error as the Dan Brown trap—maybe moreso, because at least the Dan Brown trap doesn't tell you to ignore important evidence.
If you make an argument along the lines of "the prior probability of that hypothesis is low", deduct 10 of your contrarian points. If you make a reference to the universal prior, deduct 20 points and feel guilty for the next few weeks.
Note that I think I'm a decent contrarian but I'm bad at communicating contrarian ideas; I'm not sure to what extent this is a personal quirk or a general problem when talking to people who start out assuming that you're crazy/deluded/trolling/whatever. If there is a General Contrarian Heuristic that's more amenable to communicating resultant insights then maybe that heuristic is better.
"May we not forget interpretations consistent with the evidence, even at the cost of overweighting them."
For comparison, the General Chess Heuristic: Think about a move you could make, think about the moves your opponent could make in reply, think about what moves you could make if they replied with any of those candidate moves, &c.; evaluate all possible resultant positions, subject to search heuristics and time constraints.
What's interesting is that novice chess players reliably forget to even consider what moves their opponent could make; their thought process barely includes the opponent's possible thought process as a fundamental subroutine. I think ...
I'm worried that LW doesn't have enough good contrarians and skeptics, people who disagree with us or like to find fault in every idea they see, but do so in a way that is often right and can change our minds when they are. I fear that when contrarians/skeptics join us but aren't "good enough", we tend to drive them away instead of improving them.
For example, I know a couple of people who occasionally had interesting ideas that were contrary to the local LW consensus, but were (or appeared to be) too confident in their ideas, both good and bad. Both people ended up being repeatedly downvoted and left our community a few months after they arrived. This must have happened more often than I have noticed (partly evidenced by the large number of comments/posts now marked as written by [deleted], sometimes with whole threads written entirely by deleted accounts). I feel that this is a waste that we should try to prevent (or at least think about how we might). So here are some ideas: