= 27d567bc2d48d9f40e9ccc20ea370b15)
royf comments on Fake Explanations - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (81)
AC, what you're describing here is a severe case of dĂŠformation educationnelle.
Really, I can quite understand the students... if you say "I don't know" you have a zero percent chance of getting the explanation right.
If you say "I don't know" you have a zero percent chance of getting a gold star in the idiot damned school system. But it is still the rational thing to say when, in fact, you don't know. You can easily do worse than maximum entropy if you guess at random.
Furthermore, "getting it right" by guessing the verbal phrase the teacher has in mind, even if the school system gives you a gold star for it, does not necessarily mean that you possess any anticipation-controllers. All you got right was a string of words, like guessing the passphrase to the teacher's login.
"Heat conduction" is a verbal phrase which may, for someone who knows the equations, invoke genuinely explanatory equations from memory. And for someone who knows the equations, it should be obvious that the equations do not predict the further side being warmer.
If you don't know the equations, then "heat conduction" is a verbal phrase invoking magic from the Star Trek genre. Even if the teacher says "You're right!", because there were only a limited number of phrases you learned about this semester and one of them had to be "right", you still don't know anything except an arbitrary passphrase.
If you memorized the equations but you didn't apply them, then "heat conduction" still invokes magic - it's not enough to know what symbols to scribble on the final exam, you've got to do the math or it doesn't count.
You are overstating the case by a large margin.
Saying "I don't know" may be, to a large degree, the true state of your belief when you use probability theory. But in this case it's not the rational thing to say when you use decision theory. "I don't know" is true, but it is a non-answer to the question, and doesn't get you points. It's a different matter whether this point system is effective or moral, but as long as it's there, that's what you play by.
If you base your guess correctly on an incomplete model of reality, which you've constructed correctly from past observations, you can never do worse, on average, than maximum entropy. More evidence can never lead to less information (as per the Data Processing Inequality).
On the contrary, it mean exactly that. Being rewarded for predictive powers improves your model of the world, whereas "I don't know" is an excuse for not knowing.
In fact, the mechanism employed by the teacher, for all its flaws, achieves 3 important goals:
I disagree. The proper response to not knowing the answer is to admit to not knowing and then give your best guess, not to try to hide your ignorance, because if you succeed then the teacher doesn't know you need help. A student who is more concerned with not displaying ignorance than with not being ignorant is not trying to learn, which is not rational. That which can be destroyed by the truth should be, and it probably won't be if you try to avoid finding out what the truth is.
The key phrase here is "on average". If you guess at random from all possible explanations of a given phenomenon, you will, on average, die before guessing the correct answer. There is a reason the monkeys with typewriters are given infinite time to reproduce Hamlet.
Moreover, as the set of answers considered increases in size, the expected utility from giving any one answer tends towards the expected utility of a wrong answer. As long as giving the wrong answer gives less utility than admitting ignorance, admitting ignorance is almost always the utility maximising option if you don't know.
If I write down a number and then take a number from a table of random numbers, and the numbers are the same, does this mean that I'm psychic? Because if getting the correct answer means that I have useful anticipation controllers then I must be.
"I don't know" is not an excuse for not knowing. That makes no sense whatsoever. "I don't know" is a statement about whether I know something or not, not a statement about whether I ought to know. If you can't admit fallibility then you will never learn anything.
The points you make about the benefits of testing students' knowledge are true. Unfortunately, they miss the point - while it is important not to penalise guessing incorrectly, so as not to dissuade admitting ignorance, it is much better to actively reward admitting that you have tried and failed. If a confused student does not always seek an explanation, the reward for seeking explanations isn't large enough yet. If students are content to remain ignorant, something is seriously wrong with your system for making students less ignorant.
If students could always get away with an "I don't know" they wouldn't have much incentive to learn anything.
More importantly, the school system main purpose is not to teach you just a collection of facts. It has to teach you how to behave in the world, where you often have to make choices based on incomplete information.
0 marks for "I don't know". 1 mark for a correct answer. -1 mark for an incorrect answer.
Not only is it a simple incentive system I've done exams that implemented similar systems. (Westpac math competition for example.)
That is a sensible scoring system which is in fact widely used.
Allow both an answer and a certainty.
-x points for an incorrect answer with certainty x
+2x points for the correct answer with certainty x
Alternately, +10^x points for a correct answer with certainty x, and +Log(1-x) points for the incorrect answer. This encourages an attempt to answer every question, even if the certainty is rated as 0.
Yes, I know, old post.
If you give the student -X points for an incorrect answer with certainty X, and +2X points for a correct answer with certainty X, the expected value of giving an answer and lying about its certainty as Y is (1-X)(-Y) + (X)(2Y) = 3XY - Y. If X is less than 1/3, the student should lie and claim that his certainty is 0, while if X is greater than 1/3, he should lie and claim that his certainty is 1.
I'm not going to try to find the maximum for the second version, but it should be obvious that the student is still better off lying about his true certainty. Of course, you could just avoid telling the student how you're going to grade, but the score will then just depend on how well the student guesses your grading criteria.
Neither of my described systems are ideal. Squared error works for binary questions, but it would reward "Pi is exactly 3, with 0 confidence".
Rather than allow continuous estimates of accuracy, I think that the ideal system would ask the student to provide a range of confidence, (five choices from "guessing" to "Certain", with equivalent probabilities), and an appropriate scoring rule; a guess would be penalized 0 for being wrong but gain little for being right, and going from "almost certain" to "certain" would add a small value to a correct answer but a large penalty to a wrong answer.
Having established the +points for correct and -points for wrong for each confidence description, do the math to determine what the actual ranges of confidence are, sanity check them against the descriptions, and then tell the student the confidence intervals. (Alternately, pick the intervals and terms and do the math to figure out the + for correct answer and -for incorrect answer for those intervals.)
It's hard to come up with a system where the student doesn't benefit from lying about his certainty. What you describe would fix the case from 4 (almost certain) to 5 (certain), but you need to get all the cases to work and it's plausible that fixing the 4 to 5 case (and, in general, increasing the incentive to pick 4) breaks the 3 to 4 case.
After all, you can't have all the transitions between certainty values add a small value to a correct answer. You must have a transition where a large value is added for a correct answer and your system may break down around such transitions.
The largest value would be added for the first confidence interval, which would also add the smallest cost to being wrong with that confidence.
Students who do not care about education do get away with not knowing anything. Detention is not much of a punishment when you don't show up.
It is difficult to prevent a student who cares deeply about eduction from admitting ignorance, since admitting ignorance is necessary in asking for explanations. The difficult task is persuading students who care about doing well to seek knowledge, rather than good marks. These students are not motivated enough to learn of their own accord - they never volunteer answers or ask questions openly, because they care more about not being thought ignorant (or, of course, keen) than about not being ignorant.
The point is not to allow students to "get away with" admitting ignorance. There is a vast difference between not knowing the answer and not wanting to know. Personally, I have never found it hard to tell the difference between students who don't want to know and students who don't want to be judged by their peers.
It is very rarely a bad idea to publicly admit that you might be wrong, especially when you are guessing. A school that does not teach the importance of separating your beliefs and your ego has failed miserably. Whatever else it has taught, it has not taught its students how to learn.
How true