# HonoreDB comments on SotW: Avoid Motivated Cognition - Less Wrong

20 28 May 2012 03:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

You are viewing a single comment's thread.

Comment author: 25 May 2012 07:14:56PM 1 point [-]

This seems like it'll be easiest to teach and test if you can artificially create a preference for an objective fact. Can you offer actual prizes? Candy? Have you ever tried a point system and have people reacted well?

Assume you have a set of good prizes (maybe chocolate bars, or tickets good for 10 points) and a set of less-good prizes (Hershey's kisses, or tickets good for 1 point).

Choose a box: Have two actual boxes, labeled "TRUE" and "FALSE". Before the class comes in, the instructor writes a proposition on the blackboard, such as "The idea that carrots are good for your eyesight is a myth promoted as part of a government conspiracy to cover up secret military technology" or "A duck's quack never echoes, and nobody knows why." If the instructor believes that the proposition is true, the instructor puts a bunch of good prizes in the TRUE box and nothing in the FALSE box. Otherwise, the instructor fills the FALSE box with less-good prizes. The class comes in, and the instructor explains the rules. Then she spends 5 minutes trying to persuade the class that she believes the proposition. After that, people who think she actually believes it line up at the TRUE box, and everyone else lines up at the FALSE box. Everyone who guessed right gets a prize from their box. If you guess TRUE and you're right, your prize is better than if you guess FALSE and are right. Repeat this for a few propositions, and it's at least a useful test for whether you can separate what you want from what seems plausible.

Comment author: 25 May 2012 09:34:20PM 3 points [-]

If the prize for correctly answering "true" is 10 times as good as the prize for correctly answering "false", then you really should be about 91% confident the correct answer is "false" before you give that answer.

Comment author: 25 May 2012 09:52:02PM 0 points [-]

Yup. The propositions need to be such that you can get more confident than that.

Comment author: 25 May 2012 10:02:24PM 3 points [-]

My point was that being biased to answer "true", even if "false" is more likely to be correct, is a rational strategy. This problem could be eliminated if the good effects of the correct answer being "true" were independant of getting the right answer. That is, if the correct answer is "true" you get 10 points, and if you answer correctly you get 1 point. That way, you want the answer to be "true", but it is not rational to let this have any effect on your answer.

Comment author: 30 May 2012 08:57:14PM 0 points [-]

I assumed the point was to illustrate how, given the motivation (candy), your thinking DOES end up rationally biased towards saying "true". It gives you a clear example of motivated thinking (I want to answer "True" even if that's not the correct answer) and puts you inside the experience while still being entirely aware that it's motivated thinking.

Comment author: 30 May 2012 09:37:48PM 0 points [-]

(I want to answer "True" even if that's not the correct answer)

That's not right. It is still the case that you want to answer "True" if the correct answer is "True", and you want to answer "False" if the correct answer is false. It's just that in the original formulation, the greater reward of correctly answer "True" means that you are willing to take a smaller chance of correctly answer "True" instead of a larger chance of correctly answering "False". Where in my modification, you want the correct answer to be "True", but if this actually influences whether you answer "True" or "False", you did something wrong.

Comment author: 30 May 2012 11:31:41PM 0 points [-]

you are willing to take a smaller chance of correctly answer "True" instead of a larger chance of correctly answering "False"

Sorry, that's what I meant by "I want to answer True even if it's not correct" :)

Where in my modification, you want the correct answer to be "True", but if this actually influences whether you answer "True" or "False", you did something wrong.

Yours strikes me as teaching the skill "disentangle mixed incentives" - there's an incentive to be correct, and an incentive for the correct answer to be "TRUE". There's also the skill of recognizing which of these two incentives you have control over (the former only). While these are certainly USEFUL skills, I question whether this really helps people avoid Motivated Cognition. On an abstract level, it seems like they might be related, but I don't think it's the sort of exercise that would build an intuitive sense of "Oh, wait, I'm doing motivated cognition here!"

I feel the original exercise, however, does a good job, because it puts the person in a position where they can SEE that they're doing motivated cognition. It lets them get a good look at what it FEELS like to be doing motivated cognition, what it looks like. Once they can recognize it, it's a LOT easier to fight it :)