Perhaps I don't understand what you were suggesting in the first place. I interpreted what you wrote as taking into account your calibration curve into a Bayesian update to make better calibrated predictions. But the calibration curve itself is basically the actual probability something will occur conditioned on your belief. So, you can use the calibration curve to get an idea of what your real confidence level should be. For example, if you say you are 70% confident for a certain prediction, look at your calibration curve. Let's say the curve says that 60% of the things you say will occur 70% of the time actually occur. Then you have reason to believe assigning a probability of 60% would be better.
Similar ideas are used in other contexts, e.g., here's the idea applied to scheduling.
This seems to be the most straightforward way to take into account your calibration information. Maybe I don't understand what you're suggesting, though. If you could make an example (with numbers) I'd be interested.
I suppose you're looking to make Bayesian updates more accurate now that I think about it, and I don't know if anyone has a systematic way of doing this. I have used various probability rules, Bayes among them, to calculate conditional probabilities to base my predictions on (e.g., here I demonstrate myself to be overconfident about how likely I would be to regret getting a vasectomy), but this is not feeling based, which seems to be what you are going for. I don't really trust my feelings here, but perhaps doing this would help calibrate them.
Yes, the primary benefit of calibration training for me has been that I can now say "hmm, if I "feel" this confident about something, it's 90% likely. If I "feel" this confident of something after running it through my basic calibration exercises, I'm 80% likely, etc. Also, if I'm asked to give a numerical estimate, I'm very good at giving a 90% confidence interval that is within range 90% of the time.
If you haven't used calibration training in this way, I highly recommend it.
In terms of what I'm trying to accomplish, you're right...
Hey all,
After reading "How to Measure Anything" I've experimented a bit with calibration training and using his calibration tools, and after being convinced by his data on the usefulness of calibration in forecasting for the real world, have seen a big update in my own calibration.
I'm wondering if anybody knows of similar tools and studies on calibration of Bayesian updating. Broadly,I imagine it would look like:
1. Using the tools and calibration methods I already use to figure out how the feeling of "correctness" of my prior correlates to a numerical value.
2. Using similar (but probably not identical) tools to figure out how "convincing" the new data feels correlates to specific numbers.
3. Calibrating these two numbers to bayes theorom, such that I know approximately how much to update the original feeling to reflect the new information
4. Using mmenomic or visualization techniques to pair the new feeling with the belief, so that next time I remembered the belief, I'd feel the slightly different calibration.
Anyways, I'm curious if anyone has experimented with these processes, if there's any research on it, or it has been previously experimented with on lesswrong. I'd definitely like to lock down a similar procedure for myself.
I should note that many times, I already do this naturally... but my guess is I systematically over and under update the feeling based on confirmation bias. I'd like to recalibrate my recalibration :).