That's probably closer to the truth than one might think. Once a belief system moves beyond rote memorization of its basic principles and becomes associated with other domains, non-rational beliefs can get very heavily embedded with outside belief networks. The feedback loop that can be created by having just a few anecdotal connections to an already established system would be severe.
The key factor is that, for people who are not strict rationalists already, the "correlation=causation" attitude is quite strong, so any neuronal links I make from new information to outside branches of knowledge can freely flow right back the way they came. Where the rationalist would have to find additional evidence to ingrain a belief, the fundamentalist is free to draw from his outside branches of knowledge to find reverse reinforcement to support the belief he's trying to learn.
Of course, we all do this to a certain extent, bootstrapping our new, tenuous beliefs by looking for associations we can make to older, more familiar territory. But fundamentalists can get through the neuronal rut-treading faster than rationalists, allowing a belief system to become ingrained that much faster.
Also, part of rationalists' training involves maintaining belief system elasticity, so we are ready to shift our conceptions as new information comes along. Fundamentalists, on the other hand, strive in exactly the reverse direction, wanting each neuronal pathway to be as unchanging as possible. There are two main reasons I can think of that this would be important: The one is that God's morality is eternal and unchanging, so the closer we bring our thought patterns out of that messy doubting game, the closer we come to "perfection". The other is that certain idea like adultery or homosexuality are expressly forbidden not to just do, but to even think about. What's a person to do? Well, once you hit the Stage 3 described above, your neural pathways will just naturally flow in the proscribed direction, avoiding extraneous pitfalls that you've edited out.
I remember reading something about this stage with professional chess players a long time ago--a chess master simulates less possible moves in their head than a player with only moderate experience, because past a certain stage, their brain pathways have seen enough games that the obviously "bad" moves simply drop out of their neural net.
Charlie Parker echoed a similar thing about jazz:
“You've got to learn your instrument. Then, you practice, practice, practice. And then, when you finally get up there on the bandstand, forget all that and just wail.”
Unfortunately, the same neural embedding that makes great chess players and musicians possible, also makes cults and other forms of indoctrination possible.
Good point. That's why I here argued against thinking about things too long. It's even more important the less rational you are. Before you know it, you are past the point that any evidence can convince you that your opinion is wrong.
Related to: Truly Part of You, A Technical Explanation of Technical Explanation
Partly because of LessWrong discussions about what really counts as understanding (some typical examples), I came up with a scheme to classify different levels of understanding so that posters can be more precise about what they mean when they claim to understand -- or fail to understand -- a particular phenomenon or domain.
Each level has a description so that you know if you meet it, and tells you what to watch out for when you're at or close to that level. I have taken the liberty of naming them after the LW articles that describe what such a level is like.
Level 0: The "Guessing the Teacher's Password" Stage
Summary: You have no understanding, because you don't see how any outcome is more or less likely than any other.
Description: This level is only included for comparison -- to show something that is not understanding. At this point, you have, a best, labels that other people use when describing the phenomenon. Maybe you can even generate the appearance of understanding on the topic. However, you actually have a maximum entropy probability distribution. In other words, nothing would surprise you, no event is more or less likely to happen, and everything is consistent with what you "know" about it. No rationalist should count this as an understanding, though it may involve knowledge of the labels that a domain uses.
Things to watch out for: Scientific-sounding terms in your vocabulary that don't correspond to an actual predictive model; your inability to say what you expect to see, and what you would be surprised by.
Level 1: The "Shut up and Calculate" Stage
Summary: You can successfully predict the phenomenon, but see it as an independent, compartmentalized domain.
Description: This is where you can predict the phenomenon, using a generative model that tells you what to expect. You are capable of being surprised, as certain observations are assigned low probability. It may even be tremendously complicated, but it works.
Though low on the hierarchy, it's actually a big accomplishment in itself. However, when you are at this stage, you see its dynamics as being unrelated to anything else, belonging to its own domain, following its own rules. While it might have parallels to things you do understand, you see no reason why the parallel must hold, and therefore can't reason about how extensive that relationship is.
Things to watch out for: Going from "It just works, I don't know what it means" to "it doesn't mean anything!" Also, becoming proud of your ignorance of its relationship to the rest of the world.
Level 2: The "Entangled Truths" Stage. (Alternate name: "Universal Fire".)
Summary: Your accurate model in this domain has deep connections to the rest of your models (whether inferential or causal); inferences can flow between the two.
Description: At this stage, your model of the phenomenon is also deeply connected to your model of everything else. Instead of the phenomenon being something with its own set of rules, you see how its dynamics interface with the dynamics of everything else in your understanding. You can derive parameters in this domain from your knowledge in another domain; you can explain how they are related.
Note the regression here: you meet this stage when your model for the new phenomenon connects to your model for "everything else". So what about the first "everything else" you understood (which could be called your "primitively understood" part of reality)? This would be the instinctive model of the world that you are born with: the "folk physics", "folk psychology", etc. Its existence is revealed in such experiments as when babies are confused by rolling balls that suddenly violate the laws of physics.
This "Level 2" understanding therefore ultimately connects everything back to your direct, raw experiences ("qualia") of the world, but, importantly, is not subordinate to them – optical illusions shouldn't override the stronger evidence that proves to you it's an illusion.
Things to watch out for: Assuming that similar behavior in different domains ("surface analogies") is enough to explain their relationship. Also, using one intersection between multiple domains as a reason to immediately collapse them together.
Level 3: The "Truly Part of You" Stage
Summary: Your models are such that you would re-discover them, for the right reasons, even they were deleted from your memory.
Description: At this stage, not only do you have good, well-connected models of reality, but they are so well-grounded, that they "regenerate" when "damaged". That is, you weren't merely fed these wonderful models outright by some other Really Smart Being (though initially you might have been), but rather, you also consistently use a reliable method for gaining knowledge, and this method would eventually stumble upon the same model you have now, no matter how much knowledge is stripped away from it.
This capability arises because your high understanding makes much of your knowledge redundant: knowing something in one domain has implications in quite distant domains, leading you to recognize what was lost – and your reliable methods of inference tell you what, if anything, you need to do to recover it.
This stage should be the goal of all rationalists.
Things to watch out for: Hindsight bias: you may think you would have made the same inferences at a previous epistemic state, but that might just be due to already knowing the answers. Also, if you're really at this stage, you should have what amounts to a "fountain of knowledge" – are you learning all you can from it?
In conclusion: In trying to enhance your own, or someone else's, understanding of a topic, I recommend identifying which level you both are at to see if you have something to learn from each other, or are simply using different standards.