I'm not sure what you're getting at in your first sentence. Are you saying that my feelings will be updated automatically given new evidence? This does happen to some extent, but as I've noted, I suspect that how much I change the feeling is supremely uncalibrated to an actual Bayesian update - I'd like to calibrate how much those feelings change based on evidence. There's a whole category of biases around the concept of "updating beliefs based on new information" - and I'd like to systematically reduce all those biases in one fell swoop with this training.
That's an interesting take by Nate on gaming calibration. I haven't noticed that tendency myself - mostly because I don't keep a global prediction curve for myself, but rather create a new one every time I'm doing calibration training, and only look at it a few times a session. I'd like to use prediction book more but haven't set up a habit of recording predictions in it.
I think that you can definitely hone calibration in an individual field, but in my experience (and based on the research of Doug Hubbard) there's definitely a global calibration factor. Just remembering to use the equivalent bet test and the absurdity test significantly improves my prediction. To get back to your mass example, I actually did a training session once on object weight, and another on predicting calories, and I find that even though I don't have much experience with doing either of those things, I was actually quite well calibrated. The only difference with them was that my confidence interval was significantly wider than in fields in which I'm more confident.
Perhaps I don't understand what you were suggesting in the first place. I interpreted what you wrote as taking into account your calibration curve into a Bayesian update to make better calibrated predictions. But the calibration curve itself is basically the actual probability something will occur conditioned on your belief. So, you can use the calibration curve to get an idea of what your real confidence level should be. For example, if you say you are 70% confident for a certain prediction, look at your calibration curve. Let's say the curve says that 60...
Hey all,
After reading "How to Measure Anything" I've experimented a bit with calibration training and using his calibration tools, and after being convinced by his data on the usefulness of calibration in forecasting for the real world, have seen a big update in my own calibration.
I'm wondering if anybody knows of similar tools and studies on calibration of Bayesian updating. Broadly,I imagine it would look like:
1. Using the tools and calibration methods I already use to figure out how the feeling of "correctness" of my prior correlates to a numerical value.
2. Using similar (but probably not identical) tools to figure out how "convincing" the new data feels correlates to specific numbers.
3. Calibrating these two numbers to bayes theorom, such that I know approximately how much to update the original feeling to reflect the new information
4. Using mmenomic or visualization techniques to pair the new feeling with the belief, so that next time I remembered the belief, I'd feel the slightly different calibration.
Anyways, I'm curious if anyone has experimented with these processes, if there's any research on it, or it has been previously experimented with on lesswrong. I'd definitely like to lock down a similar procedure for myself.
I should note that many times, I already do this naturally... but my guess is I systematically over and under update the feeling based on confirmation bias. I'd like to recalibrate my recalibration :).