part 2: "So what an expert rationalist should do to avoid this overconfidence trap?"
Apologies for flooding the comments, but I wanted to separate the ideas so they can be discussed separately. The question is how to avoid overconfidence, and bias in general. Picking up from last time:
If we can identify a bias, presumably we can also identify the optimal outcome that would happen in the absence of such bias. If we can do that, can't we also constrain ourselves in such a way that we can achieve the optimal outcome despite giving in to the bias? For example, David Balan referenced his own softball game, in which he swings a half-second to early and has been unable to tell himself "swing .5 seconds later" with any success. My advice to him was to change his batting stance such that the biased swing still produces the optimal outcome.
This idea of "changing your stance" is especially useful in situations in which you can't constrain yourself in other ways: in situations in which you know you will be biased and can't avoid making decisions in such situations. David would have to avoid the game altogether to correct his bias, but that's akin to saying that the dead don't commit bias: by adjusting his stance he can stay in the game AND have the right outcome.
In contrast to constraining your possible set of actions to unbiased ones [as I suggested in my other comment] the other possible way to deal with it is to set your starting point [your "stance"] such that the biased action/decision gets you to the right place.
“Everyone complains of his memory, but nobody of his judgment." This maxim of La Rochefoucauld rings as true today as it did back in the XVIIth century. People tend overestimate their reasoning abilities even when this overconfidence has a direct monetary cost. For instance, multiple studies have shown that investors who are more confident of their ability to beat the market receive lower returns on their investments. This overconfidence penalty applies even to the supposed experts, such as fund managers.
So what an expert rationalist should do to avoid this overconfidence trap? The seeming answer is that we should rely less on our own reasoning and more on the “wisdom of the crowds”. To a certain extent this is already achieved by the society pressure to conform, which acts as an internal policeman in our minds. Yet those of us who deem themselves not very susceptible to such pressures (overconfidence, here we go again) might need to shift their views even further.
I invite you now to experiment on how this will work in practice. Quite a few of the recent posts and comments were speaking with derision about religion and the supernatural phenomena in general. Did the authors of these comments fully consider the fact that the existence of God is firmly believed by the majority? Or that this belief is not restricted to the uneducated but shared by many famous scientists, including Newton and Einstein? Would they be willing to shift their views to accommodate the chance that their own reasoning powers are insufficient to get the right answer?
Let the stone throwing begin.