I'm in an ongoing conversation with a couple of LDS missionaries, and those incommunicable experiences seem to be their primary argument. They say they can't convince me Mormonism is true on their own, but if I read, and I pray, and I work, I'll just have that experience myself and I'll know it. And if I had more spare time, I would do all those things, primarily to give them Bayesian evidence that it's actually not true (I keep meaning to pick up my Book of Mormon, but, well, particle physics to read, LW posts to write, root documentation to go through, etc etc etc)
The primary argument that I have given back, though, is that religious experiences of this stripe simply aren't unique to Mormonism. Hindus, Muslims, Buddhists, Catholics, and ravers taking E have all had similar transcendent experiences. It is naive to take it as Bayesian evidence of mormonism.true.
I don't think Mormonism is true. I'm not even sure I should consider my friends' experiences evidence in favor of the proposition that Mormonism is true, especially given that I know other religions have similarly experience-laden representatives and "Alicorn's friends" isn't a group representative of the population as a whole. I do, however, suspect that they should consider it evidence - even strong evidence - to exactly that effect.
When an LW contributor is in need of an example of something that (1) is plainly, uncontroversially (here on LW, at least) very wrong but (2) an otherwise reasonable person might get lured into believing by dint of inadequate epistemic hygiene, there seems to be only one example that everyone reaches for: belief in God. (Of course there are different sorts of god-belief, but I don't think that makes it count as more than one example.) Eliezer is particularly fond of this trope, but he's not alone.
How odd that there should be exactly one example. How convenient that there is one at all! How strange that there isn't more than one!
In the population at large (even the smarter parts of it) god-belief is sufficiently widespread that using it as a canonical example of irrationality would run the risk of annoying enough of your audience to be counterproductive. Not here, apparently. Perhaps we-here-on-LW are just better reasoners than everyone else ... but then, again, isn't it strange that there aren't a bunch of other popular beliefs that we've all seen through? In the realm of politics or economics, for instance, surely there ought to be some.
Also: it doesn't seem to me that I'm that a much better thinker than I was a few years ago when (alas) I was a theist; nor does it seem to me that everyone on LW is substantially better at thinking than I am; which makes it hard for me to believe that there's a certain level of rationality that almost everyone here has attained, and that makes theism vanishingly rare.
I offer the following uncomfortable conjecture: We all want to find (and advertise) things that our superior rationality has freed us from, or kept us free from. (Because the idea that Rationality Just Isn't That Great is disagreeable when one has invested time and/or effort and/or identity in rationality, and because we want to look impressive.) We observe our own atheism, and that everyone else here seems to be an atheist too, and not unnaturally we conclude that we've found such a thing. But in fact (I conjecture) LW is so full of atheists not only because atheism is more rational than theism (note for the avoidance of doubt: yes, I agree that atheism is more rational than theism, at least for people in our epistemic situation) but also because
Does any of this matter? I think it might, because
So. Is theism really a uniquely awful example? If so, then surely there must be insights aplenty to be had from seeing what makes it so unique. If not, though ... Anyone got any other examples of things just about everyone here has seen the folly of, even though they're widespread among otherwise-smart people? And, if not, what shall we do about it?