Yup; the only Principia Mathematica I'd ever heard of was the one by Russel and Whitehead. I leveraged this shocking lack of knowledge into a guess that Newton lived after Galileo and before Gauss, and put down 10% on 1750; which by the rule of thumb HonoreDB came up with puts me right on the edge of overconfidence.
Yeah. I got all panicky when I encountered the question ("Argh! Newton! How can I have nothing memorized about someone as important as Newton!"). By somewhat similar reasoning I got an answer and assigned about 1/3 probability to my being within 15 years. I ended up within 10 years of the correct answer. By HonoreDB's rule that would be neither over- nor underconfident. But on discovering the answer I couldn't help thinking, "rats - I should have been more confident". I get a sense that thinking about scoring rules too much as a game can also lead to some biases.
Yvain's 2011 Less Wrong Census/Survey is still ongoing throughout November, 2011. If you haven't taken it, please do before reading on, or at least write down your answers to the calibration questions so they won't get skewed by the following discussion.
The survey includes these questions:
Suppose you state a p-confidence interval of ±a around your guess x of the true value X. Then you find that, actually, |X - x| = b. What does this say about your confidence interval?
As a first approximation, we can represent your confidence interval as a claim that the answer is uniformly randomly placed within an interval of ±(a/p), and that you have guessed uniformly within the same interval. If this is the case, your guess should on average be ±(1/3 * a/p) off, following a triangular distribution. It should be in the range (1/3 ± 3/16)(a/p) half the time. It should be less than 1/3(3 - sqrt(6)), or about .18, 1/3 of the time, and greater than 1-1/(sqrt(3), or about .42, 1/3 of the time.
So, here's a rule of thumb for evaluating your confidence intervals based on how close you're getting to the actual answer. Again, a is the radius of your interval, and p is the probability you assigned that the answer is in that interval.
1. Determine how far you were off, divide by a, and multiply by p.
2. If your result is less than .18 more than a third of the time, you're being underconfident. If your result is greater than .42 more than a third of the time, you're being overconfident.
In my case, I was 2 years off, and estimated a probability of .85 that I was within 15 years. So my result is 2/15 * .85 = .11333... That's less than the lower threshold. If I find this happening more than 1/3 of the time, I'm being underconfident.
Can anybody suggest a better system?