I don't think we can reasonably elevate our estimate of our own rationality by observing that we disagree with the consensus of a respected community.

But isn't Eliezer suggesting, in this very post, that we should use uncommon justified beliefs as an indicator that people are actually thinking for themselves as opposed to copying the beliefs of the community? I would assume that the standards we use to judge others should also apply when judging ourselves.

On the other hand, what you're saying sounds reasonable too. After all, crackpots also disagree with the consensus of a respected community.

The point is that there could be many reasons why a person would disagree with a respected community, one of which is that the person is actually being rational and that the community is wrong. Or, as seems to be the case here, that the person is actually being rational but hasn't yet encountered all the evidence which the community has. In any case, given the fact that I'm here, following a website dedicated to the art of rationality, I think that in this case rationality is quite a likely cause for my disagreement.

I should not be able to discredit a theory by the act of collecting all possible evidence and publishing before they have a chance to think things through.

I agree that if a piece of evidence is published before it is predicted, this is not evidence against the theory, but it does weaken the prediction considerably. Therefore, please don't publish this entire collection of all possible evidence, as it will make it much harder afterwards to distinguish between theories!