Here's my idea to get better at doing updates:
So, I think there are multiple levels here. You want to make sure you get the base rate part right. You also want to make sure that you get the update right. You can see how well calibrated you are for each. You might find that you're okay at estimating conditional probabilities, but bad at estimating the base rate, etc.
I tend not to use my old estimates as a prior. I'm not an expert at Bayesian probability (so maybe I get all of this wrong!). I interpret what I'm looking for as a conditional probability, maybe with an estimated prior/base rate (which you could call your "old estimate", I guess). I prefer data whenever it is available.
The toy problems are okay, and I'm sure you can generate a lot of them.
The vasectomy example was much less straightforward than I would have expected. I spent at least 10 minutes rearranging different equations for the conditional probability before finding one where I could get what I wanted in terms of what data I could find. The problem is that the data you can find in the literature often does not fit so nicely into a simple statement of Bayes rule.
Another example I found to be useful was computing my risk for developing a certain cancer. The base rate of this cancer is very low, but I have a family member who developed the cancer (and recovered, thankfully), and the relative risk for me is considerably higher. I had felt this gave me a probability of developing the cancer on the order of 10% or so, but doing the math showed that while it was higher than the base rate, it's still basically negligible. This sounds to me like the sort of exercise you want to do.
Hey all,
After reading "How to Measure Anything" I've experimented a bit with calibration training and using his calibration tools, and after being convinced by his data on the usefulness of calibration in forecasting for the real world, have seen a big update in my own calibration.
I'm wondering if anybody knows of similar tools and studies on calibration of Bayesian updating. Broadly,I imagine it would look like:
1. Using the tools and calibration methods I already use to figure out how the feeling of "correctness" of my prior correlates to a numerical value.
2. Using similar (but probably not identical) tools to figure out how "convincing" the new data feels correlates to specific numbers.
3. Calibrating these two numbers to bayes theorom, such that I know approximately how much to update the original feeling to reflect the new information
4. Using mmenomic or visualization techniques to pair the new feeling with the belief, so that next time I remembered the belief, I'd feel the slightly different calibration.
Anyways, I'm curious if anyone has experimented with these processes, if there's any research on it, or it has been previously experimented with on lesswrong. I'd definitely like to lock down a similar procedure for myself.
I should note that many times, I already do this naturally... but my guess is I systematically over and under update the feeling based on confirmation bias. I'd like to recalibrate my recalibration :).