In A Technical Explanation of Technical Explanation, Eliezer writes,
You should only assign a calibrated confidence of 98% if you're confident enough that you think you could answer a hundred similar questions, of equal difficulty, one after the other, each independent from the others, and be wrong, on average, about twice. We'll keep track of how often you're right, over time, and if it turns out that when you say "90% sure" you're right about 7 times out of 10, then we'll say you're poorly calibrated.
...
What we mean by "probability" is that if you utter the words "two percent probability" on fifty independent occasions, it better not happen more than once
...
If you say "98% probable" a thousand times, and you are surprised only five times, we still ding you for poor calibration. You're allocating too much probability mass to the possibility that you're wrong. You should say "99.5% probable" to maximize your score. The scoring rule rewards accurate calibration, encouraging neither humility nor arrogance.
So I have a question. Is this not an endorsement of frequentism? I don't think I understand fully, but isn't counting the instances of the event exactly frequentist methodology? How could this be Bayesian?
Morendil's explanation is, as far as I can tell, correct. What's much more interesting is that examples given in terms of frequencies is required to engage our normal intuitions about probability. There's at least some research that indicates that when questions of estimation and probability are given in terms of frequencies (ie: asking 'how many problems do you think you got correct?' instead of 'what is your confidence for this answer?'), many biases disappear completely.
To whom it may concern:
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
(After the critical success of part II, and the strong box office sales of part III in spite of mixed reviews, will part IV finally see the June Open Thread jump the shark?)