I was thinking about what's a good way to measure how well-calibrated you are. The most obvious way is to say your well calibrated if, e.g., 70% of your predictions at 70% confidence level are correct; however, that implicitly assumes your predictions are independent. You can try getting around this by making lots of predictions in different areas; however, this leaves open the possibility that you might be differently calibrated in different areas.
This essay exists as a large section of my page on predictions markets on
gwern.net: http://www.gwern.net/Prediction%20markets#1001-predictionbook-nights