= 27d567bc2d48d9f40e9ccc20ea370b15)
DavidS comments on 2014 Less Wrong Census/Survey - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (724)
I am curious what kind of analysis you plan to run on the calibration questions. Obvious things to do:
For each user, compute the correlation between their probabilities and the 0-1 vector of right and wrong answers. Then display the correlations in some way (a histogram?).
For each question, compute the mean (or median) of the probability for the correct answers and for the wrong answers, and see how separated they are.
But neither of those feels like a really satisfactory measure of calibration.
At the very least, I suspect one of the analyses will be 'bucketize corresponding to certainty, then plot "what % of responses in bucket were right?"' - something that was done last year (see 2013 LessWrong Survey Results)
Last year it was broken down into "elite" and "typical" LW-er groups, which presumably would tell you if hanging out here made you better at overconfidence, or something similar in that general vicinity.