You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Manfred comments on A note about calibration of confidence - Less Wrong Discussion

12 Post author: jbay 04 January 2016 06:57AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (35)

You are viewing a single comment's thread. Show more comments above.

Comment author: Manfred 04 January 2016 08:33:56PM *  0 points [-]

Failing to assign the correct probability given your information is a failure both of accuracy and of calibration.

Suppose you take a test of many multiple choice questions (say, 5 choices), and for each question I elicit from you your probability of having the right answer. Accuracy is graded by your total score on the test. Calibration is graded by your log-score on the probabilities. Our lottery enthusiast might think they're 50% likely to have the right answer even when they are picking randomly - and because of this they will have a lower log score than someone who correctly thinks they have a 1/5 chance. These two people may have the same scores on the test, but they will have different scores on their ability to assign probabilities.

Comment author: jbay 05 January 2016 07:23:12AM *  0 points [-]

I have updated my post to respond to your concerns, expanding on your lottery example in particular. Let me know if I've adequately addressed them.