Good post; it's useful to discuss biases that people who frequent this site are especially susceptible to. This happens in US extremist religious groups too, for example see this article about the subset of people who predicted the apocalypse last year:
It’s been noted by scholars who study apocalyptic groups that believers tend to have analytical mindsets. They’re often good at math. I met several engineers, along with a mathematics major and two financial planners. These are people adept at identifying patterns in sets of data, and the methods they used to identify patterns in the Bible were frequently impressive, even brilliant. Finding unexpected connections between verses, what believers call comparing scripture with scripture, was a way to become known in the group. The essays they wrote explaining these links could be stunningly intricate.
That intricacy was part of the appeal. The arguments were so complex that they were impossible to summarize and therefore very challenging to refute. As one longtime believer, an accountant, told me: “Based on everything we know, and when you look at the timelines, you look at the evidence—these aren’t the kind of things that just happen. They correlate too strongly for it not to be important.” The puzzle was too perfect. It couldn’t be wrong.
This suggests a possibly alternative explanation, that analytical types tend/learn to enjoy systematizing, especially on topics that will be important to others. As Cosma Shalizi says,
Now, I relish the schadenfreude-laden flavors of a mega-disaster scenario as much as the next misanthropic, science-fiction-loving geek, especially when it's paired with some "The fools! Can't they follow simple math?" on the side.
Related to: Reason as memetic immune disorder, Commentary on compartmentalization
On the old old gnxp site site Razib Khan wrote an interesting piece on a failure mode of nerds. This is I think something very important to keep in mind because for better or worse LessWrong is nerdspace. It deals with how the systematizing tendencies coupled with a lack of common sense can lead to troublesome failure modes and identifies some religious fundamentalism as symptomatic of such minds. At the end of both the original article as well as in the text I quote here is a quick list summary of the contents, if you aren't sure about the VOI consider reading that point by point summary first to help you judge it. The introduction provides interesting information very useful in context but isn't absolutely necessary.
Link to original article.
Introduction
Nerd Failure Mode
This section is the part most relevant to LessWrong:
In sum:
I bolded the note on mass literacy and participation because of the interesting historical conclusion that in the United Stated mass participation in democracy inevitably made the influence of religion on policy greater. It goes against a deep assumption shared by most educated people that "democratic elections" necessarily produce "liberal" or "secular" results. It was particularly evident among pundits and particularly easy to see as foolish with the recent upheavals in the Middle East.
This last rather minor seeming note is perhaps the most relevant part of the article for aspiring rationalist. Not only is it particularly salient for those us inclined to questioning the usefulness of the category "religion" in certain context, but because nearly all of us are not religious. Our bad axioms seem unlikely to originate directly from something like a religious texts, though obviously it is plausible many of our axioms ultimately originate from such sources.Not many of us are Communists either, but we are attracted to highly consistent ideologies. We seem likely to be particularly vulnerable to bad axioms in a way most minds aren't.
So if after some thought and examination you notice that a widely respected and universally endorsed axiom in your society has clear and hard to deny implications that are in practice ignored or even denounced by most people, you should be more willing to dump such axioms than is comfortable.