I just realized that willingness to update seems very cultish from outside. Literally.
I mean -- if someone joins a cult, what is the most obvious thing that happens to them? They update heavily; towards the group teachings. This is how you can tell that something wrong is happening.
We try to update on reasonable evidence. For example we would update on a scientific article more than on a random website. However, from outside is seems similar to willingness to update on your favorite (in-group) sources, and unwillingness to update on other (out-group) sources. Just like a Jehovah Witness would update on the Watch Tower, but would remain skeptical towards Mormon literature. As if the science itself is your cult... except that it's not really the science as we know it, because most scientist behave outside the laboratory just like everyone else; and you are trying to do something else.
Okay, I guess this is nothing new for a LW reader. I just realized now, on the emotional level, how willingness to update, considered a virtue on LW, may look horrifying to an average person. And how willingness to update on trustworthy evidence more than on untrustworthy evidence, probably seems like hypocrisy, like a rationalization for preferring your in-group ideas to out-group ideas.
I worry that this is a case of finding a 'secret virtue' in one's vices: I think we're often tempted to pick some outstandingly bad feature of ourselves or an organization we belong to and explain it as the necessary consequence of a necessary and good feature.
My reason for thinking that this is going on here is that another explanation seems much more plausible. For one thing, you'd think the effect of seeing someone heavily update would depend on knowing them before and after. But how many people who think of LW this way think so because they knew someo...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.