That is an amazing result! I guess the whole idea that losing faith is a sin, that questioning god is a sin comes in here. Also, the idea of "shunning" people who have placed themselves outside the current fantasy about what is true and false. A fierce shared belief in something stupid may be the best way to create a highly loyal organization. Perhaps the thing believed in doesn't have to be stupid, but presumably the belief in it must be far beyond what is justified by facts alone in order to have it serve any purpose of group-binding.
I know I have heard Mormons talk about their sadness over losing all their friends and family as they came close to losing their faith, or perhaps more to the piont, acknowledging out loud to themselves and perhaps others that they had lost their faith.
So what would allow a person who thinks he needs to be part of some group the freedom to consider ideas that would set him apart from that group? You'd have to make a strong case that there is some really fantastic payoff to get them to want to do that.
I recall seeing, in one of the AI-boxing discussion threads, a comment to the effect that the first step for EY to get out was to convince the other party to even play the game at all.
It has since then occurred to me that this applies to a lot of my interactions. Many people who know me IRL and know a belief of mine which they do not agree with and do not want to be convinced of often adopt the strategy of not talking with me about it at all. For me to convince one of these people of something, first I have to convince them to talk about it at all.
(Note, I don't think this is because I'm an unpleasant person to converse with. Excuses given are along the lines of "I never win an argument with you" and "you've studied it a lot more than I have, it's an unfair discussion". I don't think I'm claiming anything too outlandish here; average humans are really bad at putting rational arguments together.)
I suppose the general form is: in order to convince someone of a sufficiently alien (to them) P, first you must convince them to seriously think about P. This rule may need to be applied recursively (e.g., "seriously think about P" may require one or more LW rationality techniques).
As a practical example, my parents are very religious. I'd like to convince them to sign up for cryonics. I haven't (yet) come up with an approach that I expect to have a non-negligible chance of success. But the realization that the first goalpost along the way is to get them to seriously engage in the conversation at all simplifies the search space. (Deconversion and training in LW rationality has, of course, the best chance of success--but still a high chance of failing and I judge a failure would probably have a large negative impact on my relationship with my parents in their remaining years. That's why I'd like to convince them of just this one thing.)
I realize that this is a fairly obvious point (an application of this--raising the sanity waterline--is the point behind this entire site!), but I haven't seen this explicitly noted as being a general pattern and now that I note it, I see it everywhere--hence this post.