= 27d567bc2d48d9f40e9ccc20ea370b15)
Nanashi comments on Calibration Test with database of 150,000+ questions - LessWrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (31)
That's a fair criticism, but if we're going down this road we've also gotta recognize the limitations of a multiple choice calibration test. Both styles suffer from the "You know it or you don't" dichotomy. If these questions were all multiple choice, you'd still have the same rectangular shaped graph, it would just start at 50% (assuming a binary choice) instead of 0%.
The big difference is the solution sets that the different styles represent. There are plenty of situations in life where there are a few specific courses of action to choose from. But there are also plenty of situations where that's not the case.
But, I will say that a multiple choice test definitely yields a "pretty" calibration curve much faster than an open-ended test. You've got a smaller range of values, and the nature of it lets you more confidently rule out one answer or the other. So the curve will be smoother faster. Whereas this will be pretty bottom heavy for a while.
That means that for those questions most probabilities are either close to 0 or close to 1. This suggests that given this set of questions it would probably be a good idea to increase "resolution" near those two points. For that purpose, perhaps instead of asking for confidence levels expressed as percentages you could ask for confidence levels expressed as odds or log odds. For example, users could express their confidence levels using odds expressed as ratios 2^n:1, for n=k,...,0,...,-k.
That's an interesting thought but I do suspect that you'd have to answer a lot of questions to see any difference whatsoever. If you're perfectly calibrated and answer 100 questions that you are either 99.99% confident or 99.9% confident, there's a very good chance that you'll get all 100 questions right, regardless of which confidence level you pick.