Related to: Epistemic Learned Helplessness, Bounded Distrust

This seems like a particularly insightful essay that expresses very well an idea I've also had: the difference between people with relatively normal beliefs and people with crazy-sounding beliefs is often that the people with crazy-sounding beliefs trust different authorities, and then "rationally" construct crazy-sounding beliefs based on that trust.

A few key paragraphs:

[T]here are two very different phenomena at play here, each of which subvert the flow of information in very distinct ways. Let’s call them echo chambers and epistemic bubbles. Both are social structures that systematically exclude sources of information. Both exaggerate their members’ confidence in their beliefs. But they work in entirely different ways, and they require very different modes of intervention. An epistemic bubble is when you don’t hear people from the other side. An echo chamber is what happens when you don’t trust people from the other side.

...

Crucially, echo chambers can offer a useful explanation of the current informational crisis in a way that epistemic bubbles cannot. Many people have claimed that we have entered an era of ‘post-truth’. Not only do some political figures seem to speak with a blatant disregard for the facts, but their supporters seem utterly unswayed by evidence. It seems, to some, that truth no longer matters.

This is an explanation in terms of total irrationality. To accept it, you must believe that a great number of people have lost all interest in evidence or investigation, and have fallen away from the ways of reason. The phenomenon of echo chambers offers a less damning and far more modest explanation. The apparent ‘post-truth’ attitude can be explained as the result of the manipulations of trust wrought by echo chambers. We don’t have to attribute a complete disinterest in facts, evidence or reason to explain the post-truth attitude. We simply have to attribute to certain communities a vastly divergent set of trusted authorities.

Listen to what it actually sounds like when people reject the plain facts – it doesn’t sound like brute irrationality. One side points out a piece of economic data; the other side rejects that data by rejecting its source. They think that newspaper is biased, or the academic elites generating the data are corrupt. An echo chamber doesn’t destroy their members’ interest in the truth; it merely manipulates whom they trust and changes whom they accept as trustworthy sources and institutions.

And, in many ways, echo-chamber members are following reasonable and rational procedures of enquiry. They’re engaging in critical reasoning. They’re questioning, they’re evaluating sources for themselves, they’re assessing different pathways to information. They are critically examining those who claim expertise and trustworthiness, using what they already know about the world. It’s simply that their basis for evaluation – their background beliefs about whom to trust – are radically different. They are not irrational, but systematically misinformed about where to place their trust.