Inspired by your final paragraph, I sought out a variety of test questions on the web -- both on Steven's blog and elsewhere. I was expecting systematic overconfidence, with a smaller chance of systematic underconfidence, throughout the probability spectrum.

Instead I found a very interesting pattern.

When I was 90% or 95% certain of a fact, I was slightly overconfident. My 90% estimates shook out at about 80%, and my 95% estimates shook out around 90%. When I was completely uncertain of a fact, I was also slightly overconfident, but within the realm of experimental error.

But when I was just 50% confident of a fact, I was almost always wrong. Far more often than anyone could achieve by random guessing: my wrongness was thorough and integrated and systematic.

Clearly, that feeling of slight concern which I've always interpreted as, "I think I remember X, but it could go either way," actually means something closer to, "X is not true; my beliefs are inconsistent."

If I'm sure I know something, I probably do. If I'm sure I'm clueless, I probably am. But if I think I might know something, then I almost certainly have it backwards.

Is this a common bias which I should have read about by now?