Comment author:Jiro
17 September 2014 02:47:01PM
*
0 points
[-]

and going from "almost certain" to "certain" would add a small value to a correct answer but a large penalty to a wrong answer.

It's hard to come up with a system where the student doesn't benefit from lying about his certainty. What you describe would fix the case from 4 (almost certain) to 5 (certain), but you need to get all the cases to work and it's plausible that fixing the 4 to 5 case (and, in general, increasing the incentive to pick 4) breaks the 3 to 4 case.

After all, you can't have all the transitions between certainty values add a small value to a correct answer. You must have a transition where a large value is added for a correct answer and your system may break down around such transitions.

Comment author:Jiro
18 September 2014 01:37:24AM
0 points
[-]

That would mean a large value would be added when going from "guess" to "almost guess", which would mean that it would be beneficial for a student to lie and claim to almost guess when he's really completely guessing.

Comment author:Decius
18 September 2014 02:31:29AM
1 point
[-]

Suppose the student thinks that there is a 10% chance that he is right, and the reward structure is +5/-1 for confidence interval 1.

In fact, make the reward structure:(right/wrong) 1/0, 6/-1, 10/-3, 13/-6, 15/-10, 16/-15

That puts the breakpoints at roughly even intervals, keeps the math easy, and with a little bit of clarifying exactly where the breakpoints are, doesn't reward someone who accurately determines their accuracy and then lies about it.

Comment author:Jiro
18 September 2014 02:42:32PM
3 points
[-]

I sat down late last night trying to prove that this couldn't work and instead proved that it could. If I did this correctly, in order for it to work, with the confidences increasing from 0 to 1,

left side confidence <= (difference in Y)/(difference in X + difference in Y)

right side confidence >= (difference in Y)/(difference in X + difference in Y).

Differences in X are 5, 4, 3, 2, 1 and differences in Y are 1, 2, 3, 4, 5 leading to values of 1/6 through 5/6; as 0 < 1/6 < 1/5 < 2/6 < 2/5 < 3/6 < 3/5 < 4/6 < 4/5 < 5/6 < 1 this is immune to lying within a single interval (and also turns out to be so for multiple intervals).

Comment author:Decius
19 September 2014 06:27:02AM
0 points
[-]

So, what are the downsides of making this a grading standard? The biggest one I see is that it would be unfair except in classes that have as prerequisites an outstanding score in a class that covers credence calibration.

## Comments (84)

Old*0 points [-]It's hard to come up with a system where the student doesn't benefit from lying about his certainty. What you describe would fix the case from 4 (almost certain) to 5 (certain), but you need to get all the cases to work and it's plausible that fixing the 4 to 5 case (and, in general, increasing the incentive to pick 4) breaks the 3 to 4 case.

After all, you can't have

allthe transitions between certainty values add a small value to a correct answer. You must have a transition where a large value is added for a correct answer and your system may break down around such transitions.The largest value would be added for the first confidence interval, which would also add the smallest cost to being wrong with that confidence.

That would mean a large value would be added when going from "guess" to "almost guess", which would mean that it would be beneficial for a student to lie and claim to almost guess when he's really completely guessing.

Suppose the student thinks that there is a 10% chance that he is right, and the reward structure is +5/-1 for confidence interval 1.

In fact, make the reward structure:(right/wrong) 1/0, 6/-1, 10/-3, 13/-6, 15/-10, 16/-15

That puts the breakpoints at roughly even intervals, keeps the math easy, and with a little bit of clarifying exactly where the breakpoints are, doesn't reward someone who accurately determines their accuracy and then lies about it.

I sat down late last night trying to prove that this couldn't work and instead proved that it could. If I did this correctly, in order for it to work, with the confidences increasing from 0 to 1,

left side confidence <= (difference in Y)/(difference in X + difference in Y)

right side confidence >= (difference in Y)/(difference in X + difference in Y).

Differences in X are 5, 4, 3, 2, 1 and differences in Y are 1, 2, 3, 4, 5 leading to values of 1/6 through 5/6; as 0 < 1/6 < 1/5 < 2/6 < 2/5 < 3/6 < 3/5 < 4/6 < 4/5 < 5/6 < 1 this is immune to lying within a single interval (and also turns out to be so for multiple intervals).

So, what are the downsides of making this a grading standard? The biggest one I see is that it would be unfair except in classes that have as prerequisites an outstanding score in a class that covers credence calibration.