= 27d567bc2d48d9f40e9ccc20ea370b15)
JoshuaZ comments on PredictionBook.com - Track your calibration - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (52)
Do you have any evidence for this? I don't remember any strongly domain-specific results in Tetlock's study, the book I read about calibration in business, or any studies. Nor does Wikipedia mention anything except domain experts being overconfident (as opposed to people being random outside their domain even when supposedly calibrated, as you imply), which is fixable with calibration training.
And this is what I would expect given that the question is not about accuracy (one would hope experts would win in a particular domain) but about calibration - why can't one accurately assess, in general, one's ignorance?
(I have >1100 predictions registered on PB.com and >=240 judged so far; I can't say I've noticed any especial domain-related correlations.)
Note that there are some large classes of predictions which by nature will strongly cluster and won't show up until a fair bit in the future. For example there are various AI related predictions going about 100 years out. You've placed bets on 12 of them by my count. They strongly correlate with each other (for example general AI by 2018 and general AI by 2030). For that sort of issue it is very hard to notice domain related correlation when almost nothing in the domain has reached its judgement date yet. There are other issues with this sort of thing as well, such as a variety of the long-term computational complexity predictions (I'm ignoring here the Dick Lipton short-term statements which everyone seems to think are just extremely optimistic.). Have there been enough different domains that have had a lot of questions that one could notice domain specific predictions?
All that is true - and why it was the last and least of my points, and in parentheses even.