Your point regarding the overconfidence of most domain experts is a strong one. I've updated :) This is not quite antipodal to the incompetent most overestimating their percentile competence - D-K.
I was merely imagining, without evidence, that some of the calibration training would be general and some would be domain specific. Certainly you'd learn to calibrate, in general. You just wouldn't automatically be calibrated in all domains. Obviously, if you've optimized on your expertise in a domain (or worse: on getting credit for a single bold overconfident guess), then I don't expect you to have optimized your calibration for that domain. In fact, I have only a weak opinion about whether domain experts should be better or worse calibrated on average in their natural state. I'm guessing they'll overly signal confidence (to their professional+status benefit) moreso than that they're really more overconfident (when it comes to betting their own money).
Fortunately, Dunning-Kruger does not seem to be universal (not that anyone who would understand or care about calibration would also be in the stupid-enough quartiles in the first place).
Certainly you'd learn to calibrate, in general. You just wouldn't automatically be calibrated in all domains.
Again, I don't see why I couldn't. All I need is a good understanding of what I know, and then anytime I run into predictions on things I don't know about, I should be able to estimate my ignorance and adjust my predictions closer to 50% as appropriate. If I am ...
Our hosts at Tricycle Developments have created PredictionBook.com, which lets you make predictions and then track your calibration - see whether things you assigned a 70% probability happen 7 times out of 10.
The major challenge with a tool like this is (a) coming up with good short-term predictions to track (b) maintaining your will to keep on tracking yourself even if the results are discouraging, as they probably will be.
I think the main motivation to actually use it, would be rationalists challenging each other to put a prediction on the record and track the results - I'm going to try to remember to do this the next time Michael Vassar says "X%" and I assign a different probability. (Vassar would have won quite a few points for his superior predictions of Singularity Summit 2009 attendance - I was pessimistic, Vassar was accurate.)