I was thinking about what's a good way to measure how well-calibrated you are. The most obvious way is to say your well calibrated if, e.g., 70% of your predictions at 70% confidence level are correct; however, that implicitly assumes your predictions are independent. You can try getting around this by making lots of predictions in different areas; however, this leaves open the possibility that you might be differently calibrated in different areas.