Yes, the primary benefit of calibration training for me has been that I can now say "hmm, if I "feel" this confident about something, it's 90% likely. If I "feel" this confident of something after running it through my basic calibration exercises, I'm 80% likely, etc. Also, if I'm asked to give a numerical estimate, I'm very good at giving a 90% confidence interval that is within range 90% of the time.
If you haven't used calibration training in this way, I highly recommend it.
In terms of what I'm trying to accomplish, you're right that I want a way to make my Bayesian updates more accurate. Part of the problem in training this is that I AM a bit fuzzy on the math. Like, I mostly get it when talking about toy problems like taking balls from a cup or figuring out how likely a disease test is to make false positives, but it gets very confusing how all the numbers work when you're talking about a previous prior you had where you thought you were 30% likely to reget a vasectomy, and then you get new information that suggests the base rate is 10% of the population (and of course, that's still a relatively simple example where you have hard numbers).
My basic idea was to calibrate on toy problems, then use the same feelings based "ok, I know that this feeling correlates to "x decibels of evidence" - but I don't really have a surefire plan more than that.
Here's my idea to get better at doing updates:
So, I think there are multiple levels here. You want to make sure you get the base rate part right. You also want to make sure that you get the update right. You ...
Hey all,
After reading "How to Measure Anything" I've experimented a bit with calibration training and using his calibration tools, and after being convinced by his data on the usefulness of calibration in forecasting for the real world, have seen a big update in my own calibration.
I'm wondering if anybody knows of similar tools and studies on calibration of Bayesian updating. Broadly,I imagine it would look like:
1. Using the tools and calibration methods I already use to figure out how the feeling of "correctness" of my prior correlates to a numerical value.
2. Using similar (but probably not identical) tools to figure out how "convincing" the new data feels correlates to specific numbers.
3. Calibrating these two numbers to bayes theorom, such that I know approximately how much to update the original feeling to reflect the new information
4. Using mmenomic or visualization techniques to pair the new feeling with the belief, so that next time I remembered the belief, I'd feel the slightly different calibration.
Anyways, I'm curious if anyone has experimented with these processes, if there's any research on it, or it has been previously experimented with on lesswrong. I'd definitely like to lock down a similar procedure for myself.
I should note that many times, I already do this naturally... but my guess is I systematically over and under update the feeling based on confirmation bias. I'd like to recalibrate my recalibration :).