= 27d567bc2d48d9f40e9ccc20ea370b15)
faul_sname comments on Shut Up And Guess - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (105)
I vaguely recall reading an anecdote about a similar testing scheme where you had to give an actual numerical confidence value for each answer. Saying you were 100% confident of an answer that was wrong would give you minus infinity points.
I bet that would be even less popular with students.
I've given those kinds of tests in my decision analysis and my probabilistic analysis courses (for the multiple choice questions). Four choices, logarithmic scoring rule, 100% on the correct answer gives 1 point, 25% on the correct answer gives zero points, and 0% on the correct answer gives negative infinity.
Some students loved it. Some hated it. Many hated it until they realized that e.g. they didn't need 90% of the points to get an A (I was generous on the points-to-grades part of grading).
I did have to be careful; minus infinity meant that on one question you could fail the class. I did have to be sure that it wasn't a mistake, that they actually meant to put a zero on the correct answer.
If you want to try, you might want to try the Brier scoring rule instead of the logarithmic; it has a similar flavor without the minus infinity hassle.
What does 0.01% on the wrong answer get you?
Depends what you do with the other 99.99% and the other three answers, I assume.
In a two-answer scenario, if I'm understanding bill's version of the log scoring rule correctly, giving p=0.9999 to the right answer and p=0.0001 to the wrong answer should get you [log(0.9999)-log(1/2)]/log(2) ~= 0.99986 points. With four answers, giving p=0.9997 to the right answer and p=0.0001 to each of three wrong answers should get you [log(0.9997)-log(1/4)]/log(4) ~= 0.99978 points.