Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Jiro comments on Fake Explanations - Less Wrong

58 Post author: Eliezer_Yudkowsky 20 August 2007 09:13PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (84)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: Jiro 17 September 2014 02:47:01PM *  0 points [-]

and going from "almost certain" to "certain" would add a small value to a correct answer but a large penalty to a wrong answer.

It's hard to come up with a system where the student doesn't benefit from lying about his certainty. What you describe would fix the case from 4 (almost certain) to 5 (certain), but you need to get all the cases to work and it's plausible that fixing the 4 to 5 case (and, in general, increasing the incentive to pick 4) breaks the 3 to 4 case.

After all, you can't have all the transitions between certainty values add a small value to a correct answer. You must have a transition where a large value is added for a correct answer and your system may break down around such transitions.

Comment author: Decius 18 September 2014 01:13:42AM 0 points [-]

The largest value would be added for the first confidence interval, which would also add the smallest cost to being wrong with that confidence.

Comment author: Jiro 18 September 2014 01:37:24AM 0 points [-]

That would mean a large value would be added when going from "guess" to "almost guess", which would mean that it would be beneficial for a student to lie and claim to almost guess when he's really completely guessing.

Comment author: Decius 18 September 2014 02:31:29AM 1 point [-]

Suppose the student thinks that there is a 10% chance that he is right, and the reward structure is +5/-1 for confidence interval 1.

In fact, make the reward structure:(right/wrong) 1/0, 6/-1, 10/-3, 13/-6, 15/-10, 16/-15

That puts the breakpoints at roughly even intervals, keeps the math easy, and with a little bit of clarifying exactly where the breakpoints are, doesn't reward someone who accurately determines their accuracy and then lies about it.

Comment author: Jiro 18 September 2014 02:42:32PM 3 points [-]

I sat down late last night trying to prove that this couldn't work and instead proved that it could. If I did this correctly, in order for it to work, with the confidences increasing from 0 to 1,

left side confidence <= (difference in Y)/(difference in X + difference in Y)

right side confidence >= (difference in Y)/(difference in X + difference in Y).

Differences in X are 5, 4, 3, 2, 1 and differences in Y are 1, 2, 3, 4, 5 leading to values of 1/6 through 5/6; as 0 < 1/6 < 1/5 < 2/6 < 2/5 < 3/6 < 3/5 < 4/6 < 4/5 < 5/6 < 1 this is immune to lying within a single interval (and also turns out to be so for multiple intervals).

Comment author: Decius 19 September 2014 06:27:02AM 0 points [-]

So, what are the downsides of making this a grading standard? The biggest one I see is that it would be unfair except in classes that have as prerequisites an outstanding score in a class that covers credence calibration.