paulfchristiano:

Desire for accurate beliefs. Many people don't feel that improving the quality of their beliefs is worthwhile. This is a combination of a failure to recognize exactly how wrong many of their beliefs are and a general perception that it doesn't matter anyway.

This is a tremendously important issue, to which I think the post doesn't give the necessary attention. (It's also generally neglected on LW whenever topics like these are discussed.)

Here on LW, there is often a tacit assumption that changing one's beliefs towards greater accuracy is always a good and rational thing to do. However, this neglects two crucial factors. First, re-evaluating and changing one's beliefs has a cost in time and effort, which must be offset by some benefits to make it rational (in the economic sense at least). Second, when it comes to beliefs whose relevance is primarily of signaling (rather than instrumental) character, one may well be better off having an incorrect belief with superior signaling properties.

Therefore, when people feel that improving the quality of their beliefs is not worthwhile, there is at least a possibility that they might be right about it, and it's completely unjustified to dismiss this attitude as false or misguided out of hand.

This is related to the perception that there is no free lunch: as humans we resist the suggestion that we can get anything for free, and acting intelligently seems to be considered too easy.

But this is a very accurate heuristic. It's basically a correct application of the weak efficient markets hypothesis. When it comes to beliefs that have no instrumental implications, it's relatively easy to learn about biases from sources such as LW sequences and then use this knowledge to find and correct a bunch of incorrect beliefs. However, when it comes to beliefs that are relevant for practical action, figuring out how to improve those is extremely difficult. There is no straightforward way at all to apply LW sequence-style general intellectual skills profitably.