= 27d567bc2d48d9f40e9ccc20ea370b15)
Nanashi comments on Calibration Test with database of 150,000+ questions - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (31)
That means that for those questions most probabilities are either close to 0 or close to 1. This suggests that given this set of questions it would probably be a good idea to increase "resolution" near those two points. For that purpose, perhaps instead of asking for confidence levels expressed as percentages you could ask for confidence levels expressed as odds or log odds. For example, users could express their confidence levels using odds expressed as ratios 2^n:1, for n=k,...,0,...,-k.
That's an interesting thought but I do suspect that you'd have to answer a lot of questions to see any difference whatsoever. If you're perfectly calibrated and answer 100 questions that you are either 99.99% confident or 99.9% confident, there's a very good chance that you'll get all 100 questions right, regardless of which confidence level you pick.