a claim to actually being good at this sort of thing
If I'm reading the chart on that page correctly, Gwern is extremely well calibrated. Is the accuracy row for each confidence column telling us what fraction of predictions Gwern assigned a given confidence to have been right? He's got 50% - 44%, 60% - 64%, 70% - 71%, 80% - 83%, 90% - 92%, and 100% - 96%. That's incredible!
Is the accuracy row for each confidence column telling us what fraction of predictions Gwern assigned a given confidence to have been right?
Yes, something like that. I forget the exact details of how it bins.
That's incredible!
Thank you. That's years of practice and some useful heuristics at work there.
Disclaimer: I have not read this book. I'm posting it in the expectation that others may enjoy it as much as I'm sure I would if I had time to read it myself.
This looks interesting as an extended worked example of Bayesian reasoning (the "scientific approach" of the title).
Edited to add:
There are many signs in the above block of text that this book is not up to Lesswrong standards. As gwern suggests, reading it should be done with an adversarial attitude.
I propose some more useful goals than finding someone for whom we can cheer loudly as a properly qualified member of our tribe: find worked examples that let you practice your art; find structured activities that will actually lead you to practice your art; try to critically assess arguments that use the tools we think powerful, then discuss your criticism on a forum like Lesswrong where your errors are likely to be discovered and your insights are likely to be rewarded (with tasty karma).