When an LW contributor is in need of an example of something that (1) is plainly, uncontroversially (here on LW, at least) very wrong but (2) an otherwise reasonable person might get lured into believing by dint of inadequate epistemic hygiene, there seems to be only one example that everyone reaches for: belief in God. (Of course there are different sorts of god-belief, but I don't think that makes it count as more than one example.) Eliezer is particularly fond of this trope, but he's not alone.
How odd that there should be exactly one example. How convenient that there is one at all! How strange that there isn't more than one!
In the population at large (even the smarter parts of it) god-belief is sufficiently widespread that using it as a canonical example of irrationality would run the risk of annoying enough of your audience to be counterproductive. Not here, apparently. Perhaps we-here-on-LW are just better reasoners than everyone else ... but then, again, isn't it strange that there aren't a bunch of other popular beliefs that we've all seen through? In the realm of politics or economics, for instance, surely there ought to be some.
Also: it doesn't seem to me that I'm that a much better thinker than I was a few years ago when (alas) I was a theist; nor does it seem to me that everyone on LW is substantially better at thinking than I am; which makes it hard for me to believe that there's a certain level of rationality that almost everyone here has attained, and that makes theism vanishingly rare.
I offer the following uncomfortable conjecture: We all want to find (and advertise) things that our superior rationality has freed us from, or kept us free from. (Because the idea that Rationality Just Isn't That Great is disagreeable when one has invested time and/or effort and/or identity in rationality, and because we want to look impressive.) We observe our own atheism, and that everyone else here seems to be an atheist too, and not unnaturally we conclude that we've found such a thing. But in fact (I conjecture) LW is so full of atheists not only because atheism is more rational than theism (note for the avoidance of doubt: yes, I agree that atheism is more rational than theism, at least for people in our epistemic situation) but also because
- the readership of LW (and, earlier, of OB) is drawn disproportionately from communities that have long been atheistic (and not just because their members are supremely rational);
- theism has been used so often (here, and earlier on OB) as a canonical example of Wrongness that any theists who might have participated have either deconverted or gone away;
- any theists who remain are keeping quiet about it because they don't want to get jumped on.
Does any of this matter? I think it might, because
- it would be a shame to fool ourselves into thinking that rationality (or x-rationality) is more powerful than it really is;
- we may be scaring away people who (despite their obstinate clinging to irrational superstitions, etc., etc., etc) would be valuable here. As Eliezer has pointed out a few times, Robert Aumann is an Orthodox Jew; whatever mental contortions he may have to go through to maintain that, it seems fairly clear that if he turned up at LW wanting to contribute we would be ill-advised to drive him away. I bet he isn't the only person who manages to remain religious despite knowing a thing or two about rationality;
- we may be annoying and upsetting some readers, which is a (minor) pity in itself;
- perhaps there are other equally good (or better) Awful Examples that we aren't availing ourselves of because the habit of using god-belief for that purpose has become so ingrained.
So. Is theism really a uniquely awful example? If so, then surely there must be insights aplenty to be had from seeing what makes it so unique. If not, though ... Anyone got any other examples of things just about everyone here has seen the folly of, even though they're widespread among otherwise-smart people? And, if not, what shall we do about it?
Yes, theism is really a uniquely awful example.
Oversimplifying a little, let's divide the factors that lead to memetic success into two classes: those based on corresponding to evidence, and those detached from evidence. If we imagine a two-dimensional scattergram of memes rated against these two criteria, we can define a frontier of maximum success, along which any idea can only gain in one criterion by losing on the other.
Religion is what you get when you push totally for non-evidential memetic success. All ties to reality are essentially cut. As a result, all the other dials can be pushed up to 11. God is not just wise, nice, and powerful - he is all knowing, omnibenificent, and omnipotent. Heaven and Hell are not just pleasant and unpleasant places you can spend a long time in - they are the very best possible and the very worst possible experiences, and for all eternity. And so on; because all of these things happen "offstage", there's no contradictory evidence when you turn them up, so of course that's where they're going to end up.
This freedom is theism's defining characteristic. Even the most stupid pseudoscience is to some extent about "evidence": people wouldn't believe in it if they didn't think they had evidence for it, though we now understand the cognitive biases and other effects that lead them to think so. That's why there are no homeopathic cures for amputation.
I agree with other commentators that the drug war is the other real world idea that I would attack here without fear of contradiction, but I would still say that drug prohibition is a model of sanity compared to theism. Theism really is the maddest thing you can believe without being considered mad.
EDIT: as per requests, I've made a post about this: How Theism Works
Just to suggest another belief (and belief system) that at least approaches that frontier:
"There has not been one tax increase in history that actually raised revenue. And every tax cut, from the 1920s to Kennedy’s to ours, has produced more."
-Ronald Reagan to David Stockman and Martin Feldstein.
(It is perhaps only coincidental that the year was 1984.)
http://books.google.com/books?id=dBlELVvaj4cC&pg=PA43&lpg=PA43&dq=%22There+has+not+been+one+tax+increase+in+history%22+reagan+stockman&source=bl&ots=MbAlKL-1Xt&sig=FXF-KfDng... (read more)