Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

royf comments on Fake Explanations - Less Wrong

58 Post author: Eliezer_Yudkowsky 20 August 2007 09:13PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (84)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: royf 02 June 2012 07:13:32AM *  9 points [-]

You are overstating the case by a large margin.

[Saying "I don't know"] is still the rational thing to say when, in fact, you don't know.

Saying "I don't know" may be, to a large degree, the true state of your belief when you use probability theory. But in this case it's not the rational thing to say when you use decision theory. "I don't know" is true, but it is a non-answer to the question, and doesn't get you points. It's a different matter whether this point system is effective or moral, but as long as it's there, that's what you play by.

You can easily do worse than maximum entropy if you guess at random.

If you base your guess correctly on an incomplete model of reality, which you've constructed correctly from past observations, you can never do worse, on average, than maximum entropy. More evidence can never lead to less information (as per the Data Processing Inequality).

Furthermore, "getting it right" [...] does not necessarily mean that you possess any anticipation-controllers.

On the contrary, it mean exactly that. Being rewarded for predictive powers improves your model of the world, whereas "I don't know" is an excuse for not knowing.

In fact, the mechanism employed by the teacher, for all its flaws, achieves 3 important goals:

  • It motivates students to pay attention, raises their level of alertness, activates their brains.
  • It rewards students to engage their past observations to generate the most accurate belief they can about the right answer. In the process, they build a better model of the world, and they make their unknown unknowns a little more known.
  • By forcing students to generate a belief and commit to it before the correct answer is revealed, their hindsight bias is reduced.
Comment author: Chrysophylax 29 January 2013 10:58:09PM *  1 point [-]

But in this case it's not the rational thing to say when you use decision theory.

I disagree. The proper response to not knowing the answer is to admit to not knowing and then give your best guess, not to try to hide your ignorance, because if you succeed then the teacher doesn't know you need help. A student who is more concerned with not displaying ignorance than with not being ignorant is not trying to learn, which is not rational. That which can be destroyed by the truth should be, and it probably won't be if you try to avoid finding out what the truth is.

You can easily do worse than maximum entropy if you guess at random.

If you base your guess correctly on an incomplete model of reality, which you've constructed correctly from past observations, you can never do worse, on average, than maximum entropy. More evidence can never lead to less information (as per the Data Processing Inequality).

The key phrase here is "on average". If you guess at random from all possible explanations of a given phenomenon, you will, on average, die before guessing the correct answer. There is a reason the monkeys with typewriters are given infinite time to reproduce Hamlet.

Moreover, as the set of answers considered increases in size, the expected utility from giving any one answer tends towards the expected utility of a wrong answer. As long as giving the wrong answer gives less utility than admitting ignorance, admitting ignorance is almost always the utility maximising option if you don't know.

Furthermore, "getting it right" [...] does not necessarily mean that you possess any anticipation-controllers.

On the contrary, it mean exactly that. Being rewarded for predictive powers improves your model of the world, whereas "I don't know" is an excuse for not knowing.

If I write down a number and then take a number from a table of random numbers, and the numbers are the same, does this mean that I'm psychic? Because if getting the correct answer means that I have useful anticipation controllers then I must be.

"I don't know" is not an excuse for not knowing. That makes no sense whatsoever. "I don't know" is a statement about whether I know something or not, not a statement about whether I ought to know. If you can't admit fallibility then you will never learn anything.

The points you make about the benefits of testing students' knowledge are true. Unfortunately, they miss the point - while it is important not to penalise guessing incorrectly, so as not to dissuade admitting ignorance, it is much better to actively reward admitting that you have tried and failed. If a confused student does not always seek an explanation, the reward for seeking explanations isn't large enough yet. If students are content to remain ignorant, something is seriously wrong with your system for making students less ignorant.

Comment author: V_V 30 January 2013 12:45:38AM 0 points [-]

If students could always get away with an "I don't know" they wouldn't have much incentive to learn anything.

More importantly, the school system main purpose is not to teach you just a collection of facts. It has to teach you how to behave in the world, where you often have to make choices based on incomplete information.

Comment author: wedrifid 30 January 2013 12:52:04AM 2 points [-]

If students could always get away with an "I don't know" they wouldn't have much incentive to learn anything.

0 marks for "I don't know". 1 mark for a correct answer. -1 mark for an incorrect answer.

Not only is it a simple incentive system I've done exams that implemented similar systems. (Westpac math competition for example.)

Comment author: V_V 30 January 2013 12:58:22AM 1 point [-]

That is a sensible scoring system which is in fact widely used.

Comment author: Decius 30 January 2013 01:15:53AM 0 points [-]

Allow both an answer and a certainty.

-x points for an incorrect answer with certainty x
+2x points for the correct answer with certainty x

Alternately, +10^x points for a correct answer with certainty x, and +Log(1-x) points for the incorrect answer. This encourages an attempt to answer every question, even if the certainty is rated as 0.

Comment author: Jiro 18 August 2014 07:11:12PM *  2 points [-]

Yes, I know, old post.

If you give the student -X points for an incorrect answer with certainty X, and +2X points for a correct answer with certainty X, the expected value of giving an answer and lying about its certainty as Y is (1-X)(-Y) + (X)(2Y) = 3XY - Y. If X is less than 1/3, the student should lie and claim that his certainty is 0, while if X is greater than 1/3, he should lie and claim that his certainty is 1.

I'm not going to try to find the maximum for the second version, but it should be obvious that the student is still better off lying about his true certainty. Of course, you could just avoid telling the student how you're going to grade, but the score will then just depend on how well the student guesses your grading criteria.

Comment author: Decius 17 September 2014 04:03:39AM 0 points [-]

Neither of my described systems are ideal. Squared error works for binary questions, but it would reward "Pi is exactly 3, with 0 confidence".

Rather than allow continuous estimates of accuracy, I think that the ideal system would ask the student to provide a range of confidence, (five choices from "guessing" to "Certain", with equivalent probabilities), and an appropriate scoring rule; a guess would be penalized 0 for being wrong but gain little for being right, and going from "almost certain" to "certain" would add a small value to a correct answer but a large penalty to a wrong answer.

Having established the +points for correct and -points for wrong for each confidence description, do the math to determine what the actual ranges of confidence are, sanity check them against the descriptions, and then tell the student the confidence intervals. (Alternately, pick the intervals and terms and do the math to figure out the + for correct answer and -for incorrect answer for those intervals.)

Comment author: Jiro 17 September 2014 02:47:01PM *  0 points [-]

and going from "almost certain" to "certain" would add a small value to a correct answer but a large penalty to a wrong answer.

It's hard to come up with a system where the student doesn't benefit from lying about his certainty. What you describe would fix the case from 4 (almost certain) to 5 (certain), but you need to get all the cases to work and it's plausible that fixing the 4 to 5 case (and, in general, increasing the incentive to pick 4) breaks the 3 to 4 case.

After all, you can't have all the transitions between certainty values add a small value to a correct answer. You must have a transition where a large value is added for a correct answer and your system may break down around such transitions.

Comment author: Decius 18 September 2014 01:13:42AM 0 points [-]

The largest value would be added for the first confidence interval, which would also add the smallest cost to being wrong with that confidence.

Comment author: Jiro 18 September 2014 01:37:24AM 0 points [-]

That would mean a large value would be added when going from "guess" to "almost guess", which would mean that it would be beneficial for a student to lie and claim to almost guess when he's really completely guessing.

Comment author: Chrysophylax 30 January 2013 06:51:34PM *  3 points [-]

Students who do not care about education do get away with not knowing anything. Detention is not much of a punishment when you don't show up.

It is difficult to prevent a student who cares deeply about eduction from admitting ignorance, since admitting ignorance is necessary in asking for explanations. The difficult task is persuading students who care about doing well to seek knowledge, rather than good marks. These students are not motivated enough to learn of their own accord - they never volunteer answers or ask questions openly, because they care more about not being thought ignorant (or, of course, keen) than about not being ignorant.

The point is not to allow students to "get away with" admitting ignorance. There is a vast difference between not knowing the answer and not wanting to know. Personally, I have never found it hard to tell the difference between students who don't want to know and students who don't want to be judged by their peers.

It has to teach you how to behave in the world, where you often have to make choices based on incomplete information.

It is very rarely a bad idea to publicly admit that you might be wrong, especially when you are guessing. A school that does not teach the importance of separating your beliefs and your ego has failed miserably. Whatever else it has taught, it has not taught its students how to learn.

Comment author: whowhowho 30 January 2013 01:24:43AM 0 points [-]

If you guess at random from all possible explanations of a given phenomenon, you will, on average, die before guessing the correct answer

How true