I'm not sure that lack of noticing effects like this is an indication that they aren't there.

I'm aware of the possibility, and I have also mentioned it in the facebook debate. Or, more likely, I have problems finding the right words to express what I want to say:

I had situations where I didn't know something, when I forgot things, when I believed an information that was wrong, etc. Lots of them. Still doing it. Most likely will always do.

In the past (before finding LW) I have repeatedly experimented with belief in belief (because I wanted the placebo effects or social approval), but those experiments were always half-assed and very short-termed; they felt incompatible with my personality. I couldn't stop being aware that I am merely acting.

I also fail a lot at instrumental rationality. I am aware of what I should do... and I somehow just don't do it.

But I don't remember having a situation where I enjoyed being wrong or didn't care about being wrong, like described here and here. That just feels completely strange to me. I have problem empathising with people who, upon learning that they were wrong, just don't give a fuck.

Therefore -- that's why I mentioned it in the debate -- I have no clue about what to tell them to help them change their ways. I have never been there (as far as I know), and I have no idea what it feels like to be there. So I have no model that would help me test which ideas might be attractive enough to draw a person out of there.

EDIT: I feel like I should add so many disclaimers here. I am happy that at least Gleb understands what I was trying to say.

Of course there are reasons when you want to keep a map despite knowing it is not correct. When it is a useful simplification, like Newtonian physics. I am talking about people whose maps are not even approximately correct, but they still keep them because... I am only guessing here... they still provide emotional comfort.

I don't feel comfortable with having an obviously wrong map, even if it would be socially approved. I have problem belonging to most groups, because sooner or later there is a shared group map you have to accept. For example, having a political opinion (in the sense of: completely buying a standardized map) feels like insanity; on the same level as belonging to a cult. (I am strongly sympathetic to the libertarian ethics of not initiating force. That doesn't convince me that the best way to organize a society is to dismantle all states and let the warlords fight it out in the "free market".)

There may also be unlucky situations where I am wrong, other people are right, but they lack the right words to convince me (sometimes because they themselves believe the right thing for the wrong reasons, e.g. because it is a standard belief in their social group). But I don't have an epistemic strategy for avoiding such situations without making things worse on average; or course believing everything wouldn't be an improvement.

Etc.