= 27d567bc2d48d9f40e9ccc20ea370b15)
ESRogs comments on Book: AKA Shakespeare (an extended Bayesian investigation) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (13)
(Okay, I'll unpack the implication:) Assigning a probability of 10^(-3) would mean being really, really, really, really sure that the hypothesis is wrong. To be well-calibrated, you would have to be able to make ten thousand similar judgments with similar strengths of evidence and only be wrong about ten times, and if you can do that, you're very good at this sort of thing.
Assigning 10^(-13) -- i.e., suggesting that you're so good that you can do this and only be wrong one in ten million million times -- is just obviously wrong.
So I was implying that the fact that the book suggests that this kind of number can be a plausible outcome means that it isn't a very good place to learn the art of making Bayesian probability estimates. To learn to make well-calibrated estimates, I should try to learn from people who stand a snowball's chance in hell of making such estimates themselves.
For an example from someone who has a claim to actually being good at this sort of thing, see Gwern's Who wrote the Death Note script?.
If I'm reading the chart on that page correctly, Gwern is extremely well calibrated. Is the accuracy row for each confidence column telling us what fraction of predictions Gwern assigned a given confidence to have been right? He's got 50% - 44%, 60% - 64%, 70% - 71%, 80% - 83%, 90% - 92%, and 100% - 96%. That's incredible!
Yes, something like that. I forget the exact details of how it bins.
Thank you. That's years of practice and some useful heuristics at work there.