SotW: Avoid Motivated Cognition

[-]Kaj_Sotala14y200

This often works for me: think of some smart, critical and rational friend that you have, and then imagine/visualize presenting your argument to them. Or suppose that you just put up your argument on the Internet to be critiqued. Is any part of your reasoning such that you'd actually prefer not to present it in public, knowing that it won't hold up to scrutiny?

For me at least, if I imagine presenting my reasoning to someone else, I suddenly become a lot more conscious about its weak spots. And then I try to remind myself that if I can't think of a reason why those weak spots should hold up to scrutiny, my instinct should be to abandon the argument instead of just hoping that those problems won't come up.

[-]Ezekiel14y190

Anecdotal supporting evidence: In my last days as a religious person, I found myself imagining myself presenting my pro-religion arguments to the author of the Sequences, and literally could not fantasize a scenario where he found them convincing.

[-]JGWeissman14y150

"Well," says Lucy, "I suppose it could have played a role in suggesting that I eat a whole chocolate cake, but the reason why I decided to do it was to support the sugar industry. Lots of people have jobs in the sugar industry, and they've been having some trouble lately."

The obvious response to Lucy is "Is buying and eating that whole choclate cake really the best way to help the sugar industry? How does it compare to other strategies, like directly giving them money?"

The generalization is that when you say you are taking action A in pursuit of goal G, ask what other actions might be more effective at achieving G.

8TheOtherDave14y

Yeah. I also find that getting into the habit of imagining my response to the putative cause turning out not to be true can help. That is, suppose I suddenly discover that the sugar industry is actually doing really well, but the broccoli industry is in deep financial trouble. Does my decision about the cake change in response? If not, it's likely that putative cause isn't actually causal.

1JGWeissman14y

This is also a good response, though outside the pattern I described. It's generalization seems to be that when you are taking an action to help some group of people, ask would you take actions that similarly help some other group of people. So it is a somewhat narrower technique, applying when the goal is helping some group of people. Perhaps you could generalize it further.

3TheOtherDave14y

(nods) I would say, rather, that the generalization is that when I believe I'm taking A in response to condition C, ask whether my desire to do A would vary if C were radically altered. If changes in C don't correspond to changes in A, it's likely that A is not actually a response to C.

[-]daenerys14y130

Set your baseline to always assume you are rationalizing, and then have to prove that you aren't, rather than vice versa.

Something I do, that I'm surprised I don't see mentioned here, is to just assume that any point I am trying to make, or anything I think, is a rationalization (which is pretty likely).

So instead of having to think "Am I rationalizing?" (which can be hard to figure out), I change my baseline, to ALWAYS assume "I am probably rationalizing. How am I rationalizing?" and go from there. Sort of a quick run-through of what biases and semi-hidden desires could be influencing my decisions or statements at any given time. From there I can either accept or reject these "rationalizations".

This also ends up leading to many disclaimers in conversations, as mentioned in the OP. (i.e. "Well I can't know for sure what I HAD thought, because by now hindsight bias has taken hold...." or "Well, I'm completely anchored off your estimate now, but...") I see "Conversation Disclaimers" themselves to be a major skill. Maybe an exercise could be made out of that?

Quick idea: Have people in pairs have a conversation about a debatable subject. Every sentence they say has to be pre-empted with a disclaimer.

(Note: This is my immediate reaction to this post. I'll give it more thought later.)

7daenerys14y

Exercise Idea- Rationalization Listing Ask leading questions, or have everyone come up with a list of premises they think are true. (Examples- It's good to be vegetarian/paleo/omni; Political Belief System X is strongest; I enjoy Activity B; Continuing grad school is good/bad idea; etc) Once they have developed this list, have them list as many reasons as they can that they are actually just rationalizing. (i.e. have them ASSUME they are rationalizing, and then have them list ways this could be possible) The person who comes up with the MOST, wins. Example (This is a real one for me)- Premise: It's good to be vegetarian Think: "I am probably rationalizing. How? Why? Opposing evidence? Possible rationalizations: 1) I've been a vegetarian for a long time, so consistency bias wants me to continue to be one 2) Admitting I'm wrong would mean that I've been wrong for the past 8 years. 3) Being a vegetarian allows me to hold moral high ground, despite actual morality, or lack thereof, of the choice. 4) Deciding that vegetarianism is NOT a moral choice would mean that I have no reason to remain one. It's easier for me at this point to maintain the status quo, than to switch my diet back to omnivorous. 5) Social pressure- People I like and respect think that vegetarianism is a good choice, even if they themselves aren't. Arguments against: 1) Animals' life in nature is also completely horrendous. Factory farmed animals may actually have it better off than the average animal in nature. 2) Animals are not sentient enough that I should care overly much for their well-being. 3) Meat is yummy (or at least it was at one point...Now I don't much like the smell anymore) At the end, they can pair up and try to add MORE rationalization possibilities to their partner's list.

0pnrjulius13y

I'm concerned that this technique will just make people able to come up with lots of bad reasons to do things, thus making them better at rationalization. I feel like we would be better off encouraging people to come up with good reasons, and then perhaps comparing them to bad reasons.

[-]Vaniver14y130

Our conceptual understanding of 'motivated cognition', and why it's defective as a cognitive algorithm - the "Bottom Line" insight.

"Defective" isn't quite enough; you want a prescription to replace it with. Saying "this is a bad habit" seems less useful than saying "here is a good habit."

There are two obvious prescriptions I see: provide correct rationales for decisions, or do not provide rationales for decisions. Which prescription you shoot for has a radically different impact on what exercises you do, and so should be talked about quite a bit. It may be desirable to try and wipe out rationalization, first, and then progress to correct rationales.

One exercise might be asking "who will this convince?" and "whose desires do I want to maximize?". Lucy probably doesn't actually expect Marvin to be swayed by the plight of Big Sugar, and probably doesn't actually suspect that Marvin will believe she's motivated by the plight of Big Sugar, and so that deflection may be the wrong play because it's incredible.

It seems to me that social incentives will swallow most internal incentives here. If I can get more out of others by ra... (read more)

[-]Eliezer Yudkowsky14y110

"Break down what your parts have to say into parts" would be an interesting counter to rationalization - I think I'll have to call this an immediate $50 award on the grounds that I intend to test the skill itself, never mind how to teach it.

2Shmi14y

I thought that was your reason for writing the HJPEV's internal 4-way dialog.

0Vaniver14y

Awesome!

[-]HonoreDB14y120

My girlfriend says that a common case of motivated cognition is witnesses picking someone out of a lineup. They want to recognize the criminal, so given five faces they're very likely to pick one even if the real criminal's not there, whereas if people are leafing through a big book of mugshots they're less likely to make a false positive identification.

She suggests a prank-type exercise where there are two plants in the class. Plant A, who wears a hoodie and sunglasses, leaves to go to the bathroom, whereupon Plant B announces that they're pretty sure Plant A is actually $FAMOUS_ACTOR here incognito. Plant A pokes his head in, says he needs to go take a call, and leaves. See who manages to talk themselves into thinking that really is the celebrity.

[-]Emile14y100

OK, here's an exercise that could at least help noticing how motivated cognition gives us incorrect beliefs; it's like a mix between calibration and paranoid debating.

Requirements

This exercise requires a set of "interesting questions" with a numerical answer, for example "How many people were killed by guns in New York in 1990-1999?" or "What is the unemployment rate in China" (the questions should be at least related to political/social issues, no "What's the population of Mozambique").

It is also best done in a classroom-type place with a big video projector, and a bunch of computers with internet connections. Somebody who knows Excel should also need to prepare a special spreadsheet.

Step one: Crafting Arguments

Students are together in a room, with one computer each (or they can take turns using a computer, timing isn't critical); an organizer goes to each student and gives him a paper with the question, then flips a coin and announces "high!" if it's heads, and "low!" if it's tails.

Each student then has 30 minutes to prepare a set of arguments for why that particular value is high, or low. The result should be one powerp... (read more)

[-]casualhero14y90

We might also need an exercise just for getting people to understand the concept of motivated cognition at all.

"Motivated cognition" in the first place seems like a poor label because most thinking is motivated. It's redundant and arguing against "motivated cognition" at first glance sounds like arguing against any kind of motivated thinking. That's problematic because good thinking is also motivated, i.e. "I'll invent FAI because it would help the world."

One interesting thing I've heard repeated and found to be true about... (read more)

6billswift14y

I would like to suggest "motivational bias" as an alternative name; it is a more accurate description than "motivated cognition", which is much too general. I don't see spending too much time investigating things you don't want to be true as too bad a problem, a bit wasteful, but that's it. "Motivated stopping" is more likely to lead you astray, you need to remember to keep questioning things you agree with. In fact, this is a common bias that is often used by con artists, for example, and a good rule is a generalization of the common self-defense against con artists contained in the phrase, "If it looks to good to be true, it probably is." If you are immediately tempted to accept or agree with an argument, take a second look.

0thomblake14y

Interesting and useful if true

[-]Shmi14y70

how to verify that a skill has been acquired

I suppose that one way to test #2 and #3 is to ask the participants (informally, so it does not appear like a test) how useful or successful they think some other skill learning activity was. This is probably a part of the procedure already. There is a significant pressure to say that "they learned a lot and are now better at it", due to several biases, including motivated cognition. When asked to elaborate, they are likely to put forth a number of arguments. To a trained eye, it should be easy to s... (read more)

[-]Grognor14y70

One of the motivations of motivated cognition is consistency. People want to be predictable and they want to be seen as stable. So I suggest demotivating it. Have people read chapter 3 of Cialdini's Influence. I particularly like the Emerson quote:

A foolish consistency is the hobgoblin of small minds.

Yes! Teach that preferring consistency over accuracy is low-status!

I predict this will have no effect unless coupled with the following technique:

Detachment. You are not your beliefs; you are not your actions: When the opportunity to think arises, within f... (read more)

0thomblake14y

I always make sure to point out, to those who hear that quote but have not studied Emerson, that "foolish consistency" does not mean "consistency that is foolish" but rather "consistency, which is foolish".

0hamnox14y

muses I wonder how this might combine or clash with non-conformity training. Sticking to your guns against public ridicule is a form of consistency, and it does have the danger of running at cross purposes with the virtue of lightness... Can you really teach being ready to turn that dedication on a dime if the evidence blows that direction? Being faithful to THE truth instead of A truth? This suddenly sounds like a much harder task.

-2pnrjulius13y

Personally I just think Emerson often contradicted himself and didn't want to bother correcting his mistakes, so he came up with a way of making self-contradiction seem deep.

[-]Furslid14y60

One of the key markers of rationalization I've seen is rationalizations ignore tradeoffs and other option. This is obviously true only about the rationalizations about actions and policies. For instance "I want to eat the whole cake to help the sugar industry..." never finishes "...and this help to the sugar industry is worth any ill health effects." or "...and this is more efficient than other ways to help the sugar industry,"

One activity that might help is to give people a plausible proposition to argue for in their own li... (read more)

[-]beoShaffer14y60

I notice that you don't mention any existing studies. Is there a reason for this? A fairly cursory search (no backtracking of citations, or significant effort to use synonyms), brings up several relevant articles (mostly paywalled) and a book. It doesn't seem like a a super well studied field, but I don't see why its being completely ignored. The (semi) relevant stuff I found.

The Human Brain as an Evolved Rationalization Machine http://www.epjournal.net/wp-content/uploads/EP102934.pdf

Rational processing or rationalization? The effect of disconfirming i... (read more)

[-]Xom14y50

On the other hand, LW!Hermione failed to reproduce this experiment

I'm not sure how wide of an audience this post is targeting, but the !-notation feels gratuitous here. How about:

On the other hand, Hermione (LW user) failed to reproduce this experiment

[-]Morendil14y50

The title strikes me as slightly problematic: I think of skills as (positive) ways to achieve outcomes, and framing "avoidance" of some mistake as a skill doesn't feel quite right.

0The Dao of Bayes14y

Agreed - when you state it that way, it makes me realize that I don't know what the ideal state, the opposite of motivated cognition, actually looks like, or how to refer to that directly. Which then makes me suspect that simply having a clear modelwould help me build the skill ("apathetic cognition" is a fun reversal of the name :))

[-]JoachimSchipper14y50

Rationalizing anything: ask participants questions like "why did you have that cake?" or even "yesterday, you hit your kid. Why?". These questions should not be based on reality, and the instructor(s) should probably do the first two or three rounds themselves to get people used to these kinds of questions. Despite the fact that the facts are blatantly false, the participant should come up with a reason that sounds as impressive as possible. ("Yes, I hit my child yesterday. Fourteen generations of Smiths have been brought up that w... (read more)

[-]Shmi14y40

I am somewhat surprised that I cannot find the honest Devil's advocacy anywhere on the list of techniques of spotting rationalizations. I understand that EY dislikes it, because it's easy to "invent arguments for anything", but how easy is it to invent a good argument against something you deeply believe in? And by "good" I mean an argument that does not appear silly at the first or even second glance (so, no "chocolate cake in the asteroid belt" nonsense). Or maybe this is covered by the "The world is not like X and I be... (read more)

3wedrifid14y

Is this intended to be rhetorical? The literal answer seems to be "very easy, particularly with practice". Some people really are skilled at bullshit. Typically here I refer to MENSA mailing lists for examples.

0The Dao of Bayes14y

The ability to create arguments for either side is fundamental to "debate club" and what passes for national news media in the United States. Indeed, both groups are quite skilled at insisting that all issues have exactly two sides, which are completely equal in merit. I can't see much reason to value the exercise in light of such an astonishingly BAD track record... I do believe it's important to see how easy it is to rationalize something, it's just not useful to practice (unless you're looking to do writing or some other endeavor where "what if" scenarios are actually an important skill) Speaking from my own personal experience, I can explain away ~90% of plot holes in kids cartoons as being completely reasonable. (Percentage calculated based on my friends having genuine "okay, wow, that makes sense - you're so smart!" reactions.)

-3private_messaging14y

if it is easy to invent arguments for anything, then your definition of what constitutes an argument is too broad and includes nonsense.

6Richard_Kennaway14y

It's easy to invent bad arguments for anything, including arguments that, when you are motivated to produce them, you may not notice are bad arguments. Limiting "argument" to mean good arguments is too narrow a definition. As always, the definition of "argument" is not the point here.

[-]JGWeissman14y30

Val also used an upside-down W-diagram with the two worlds at the top and the four beliefs at the bottom, to emphasize the idea that the world is there first, and is fixed, and we have only a choice of what to believe within a fixed world, not a choice of which background world to live in.

I don't know what a "W-diagram" is (and a simple Google search didn't help), so I don't see how this works. Perhaps a picture could help explain.

2TheOtherDave14y

I don't know either, but I observe that an upside down W resembles a segment of a conditional tree with two points at the top and either three or four at the bottom, which sounds similar to what the OP is describing.

3CuSithBell14y

... an M...?

6TheOtherDave14y

Why yes! Also a sigma on it's side.

4tgb14y

I found the above two comments particularly funny because I had just spent the last couple minutes reading this with the vague feeling that "an upside-down W" would be better described as some other letter but I had been continuously procrastinating actually figuring out what that letter was.

2JGWeissman14y

Ah, I think I see it. So on top it has the possible worlds, and on the bottom each world branches into the possible consequences in that world of having each of the possible beliefs.

0Mercurial14y

Yep.

0[anonymous]14y

An upside down W looks like an M to me.

1Mercurial14y

To clarify, I actually think of it with two upside-down 'V' shapes. I imagine one off to my left in the world where X is true and observe the two possible outcomes based on what I believe is true in that world, and then I look off to my right to see the upside-down 'V' in the world where X is not true and consider the alternatives. I should also add that I have to put all four representations in near-mode. I view this whole process as a way of getting my brain to emotionally get that yes, it's better to have true beliefs even if they describe a world I'd rather not be in. (To clarify, in case this isn't obvious: I'm Valentine.)

7Alicorn14y

I think you should just register the username "Valentine" and use it, now before you acquire more comment history under this name.

[-]Valentine14y270

Point taken. Done.

(I had been quietly hoping to see if it's possible to change my account name from "Mercurial" to "Valentine," but I planned on doing this in the ephemeral land of "later." This just works. Thanks for pointing that out!)

1Kevin14y

Pretty sure that the software doesn't allow for name changes, but an admin can give you karma to a new account if you really want it.

8Valentine14y

I don't think I care that much about karma except for the "can post articles" limit, and that doesn't seem too difficult to reach.

[-]pnrjulius13y20

A mantra I use: "I want things to be a certain way. But I don't want to believe they are. I want to believe what is true."

A little more explanation: There was a time when I would say to myself things like, "I want to think that people are basically good." or "I want to believe that my life has not been a mistake." But now I realize that what I want is actually for people to be basically good; I don't want to think that, not if it isn't true. I don't want to believe that my life has not been a mistake; I want my life to not be ... (read more)

[-]palladias14y20

The Retrench or Retreat Game

You're making a decision and want to check if you're rationalizing. Imagine a friend or someone else whose opinion you respect raises a criticism of/question about your stated reason. Which feels easier?

Falling back to a different justification
Arguing with the friend to defend yourself

Maybe this is better as a "True Rejection" exercise, but if you find it a lot easier/more comfortable to shift arguments, your surface answer may have been a rationalization.

[-]Stefie_K14y20

The original post mentions some techniques for getting people to avoid rationalizing once they've realized they're doing it, but an earlier step is to get them to realize that they're doing it.

The key to this may be that a person who is rationalizing without realizing it is arguing with him/herself without realizing it, since it's easier to recognize (and to accept) that you're arguing than that you're rationalizing. Accordingly, getting people to realize that they're rationalizing would involve getting them to realize that they're the one that they're ar... (read more)

0Stefie_K13y

To follow up on my post: The original post talks about noticing flinches and attachments, which is similar to what I discussed above. However, I would expect it to be a lot more difficult to notice the flinch itself than it would be to notice the aftereffects, because the flinch is one moment, and the aftereffects last. (At least, when I catch myself doing it, the flinch is a single moment, and then the rationalization normalizes very quickly unless I act to counter it.) The momentary nature of the flinch would not only make it harder to notice, but also more difficult to teach people to notice flinches. There may well be a better approach to this than the one I suggested, but I have to think that exercises/activities that focus on the aftereffects would work better than ones that depend on catching that flinch.

[-]chepin14y20

EXERCISE

Ask the person to do an unpleasant task (dishwashing standing on slippery floor), which will cause the unconscious desire to finish the task.

While the task is executed, the person answers questions which can be TRUE or FALSE, the task only ends if the person complete a certain number of TRUE responses, but if the answers are wrong, it decreases the score for the task.

0pnrjulius13y

What do you mean by that?

0chepin13y

The comment was modified, thanks for the observation

[-]SarahNibs14y20

I don't always have a problem with motivated cognition, but when I do, my brain usually makes it some or all of the way through the following steps:

Notice that I'm becoming more comfortable with (feeling safer about) a decision or action I'm about to make or just made.
Notice that the physical cause of my comfort is that I recently had a thought consisting of a reason the decision could have a good outcome.
Some brain process that I haven't pinned down yet that feels like and possibly is a mix of noticing optimization by proxy, feeling disdain for no

... (read more)

3SarahNibs14y

Check Yourself For BS Practice applying your bullshit detector on your own thoughts. The key observation is that it's not important to have bullshit thoughts on which to practice detecting bullshit - like asking yourself whether you're dreaming while awake to induce lucid dreams, the practice is having a mode where your bullshit detector is on and applied to your thoughts, so that it will actually fire when the time comes. So ask yourself, over and over, a) what your last thought was, and b) whether any part of it would have been flagged as BS if you saw the thought in someone else. Exercise: Think about something, trying to come up with the right answer, maybe where you've previously stated your guess at the right answer. At some point the facilitator will ring a bell or an alarm will sound or something. Locate your previous thought (if this is hard, practice this skill). Ask whether it was bullshit. Repeat 7 times. Have a cue to do the same during the day, like whenever you hear anyone's cell phone ring or make an "I have a text!" noise.

2SarahNibs14y

Desire Generalizable Decision Processes Have everyone read Kahneman's rant on picking the action with the best expected outcome in Thinking, Fast and Slow (chapter 31, Risk Policies, especially the sermon). Encourage people to play enough poker and read enough poker theory to become at least close to neutral-EV in Vegas. Experiencing the concept of pot odds in a real game was my strongest "passing up positive-EV moves is leaving money on the table no matter how loss-averse you are" learning moment. One exercise might be to (several times) present a story in which someone makes a decision and ask participants to a) make up a near-mode explanation of why the decision feels legit, b) give a far-mode explanation of why the reasoning that led to the decision would be disastrous if everyone used it, and c) figure out what the broadest set of people/circumstances is that would allow that reasoning to be generalized and still work well. Concrete example: Holden focuses on charities that are neglected by traditional funding. a) This is great because his marginal actions will actually be at the margin, not pushed back by some giant philanthropist who suddenly funds the whole charity. b) This is awful because if everyone focused on neglected charities, the most valuable charities would receive less than optimal focus. c) As long as very few people are actually applying this heuristic, there's no danger of high-profile valuable charities suddenly losing all of their focus. So if Holden makes his decisions according to c), he's doing great, but if just a), then his algorithm is flawed though its output might be correct in this case.

1SarahNibs14y

Noting emotions. Name your emotions, simply and without judgment, as you see them arise. Inspired by Mahasi noting, it may be enough of an exercise to simply state out loud a word describing any emotion you notice, taking care to not care whether your word is perfectly accurate but instead only care about having noticed and said something with the intent that if you knew the perfect descriptor word you would have used it. For ease it may well be enough to simply say a word that means "I noticed and attended to an emotion just now". As you practice this, if you enjoy noticing the emotion regardless of the emotion itself, with any luck you'll begin to notice your emotions and thus the "suddenly I'm more comfortable and feel safer" emotion without conscious effort. As an exercise, perhaps put on a particularly emotional-roller-coaster piece of instrumental music, stopping it at random points and asking everyone to name their emotion and the emotion they were feeling due to the music just before (presumably the in-the-moment emotion will often be annoyance due to the music being interrupted).

0SarahNibs14y

Notice Rationalization Directly If you have sufficient skill at noticing rationalization such that you recognize it maybe 1+ times per day, here is an exercise that User:thejash had me do which helped my brain latch on to recognizing rationalization much more regularly: carry a pencil and a bit of paper or notebook. Every time you notice yourself rationalizing, grin and make a mark on the paper. Every time, no exceptions, for an entire week. That's all. If you must, pretend like you'll use the data for something, but it's the noticing practice itself that is useful.

0SarahNibs14y

Avoid Optimizing Proxy Measures I have a gut-level averse reaction to being comforted by a reason my decision or action may work out when I only saw that reason in order to feel safer. Here is a possible exercise to show students that optimizing proxy measures is a terrible general algorithm: hand out a worksheet of two questions to each student, asking them to answer individually. There are two possible worksheets. The first says "1. Describe in a short paragraph Vatican City. 2. Write any short paragraph you like containing the following concepts: bright, cyclic, white, round, and dappled." The second says "1. Describe in a short paragraph the appearance of the moon. 2. Write any short paragraph you like containing the following concepts: tiny, historic, beautiful, religious, gardens." Then score every paragraph by giving each a score from 0 to 1 based on how well they mentioned each of the 5 concepts, and a final score from 0 to 5 by summing the scores, and note how the proxy measure is reasonable at picking out good descriptions (is it? playtest) if you don't optimize for the proxy measure but terrible if you do.

0SarahNibs14y

Trace emotion causes. Magically acquire a gut-level realization that emotions are physical events in your brain and those events have physical causes (which may be other emotions or thoughts, since they're physical). I acquired this gut-level realization during the 2011 minicamp from User:Academian. Then, when you do manage to notice an emotion and have a moment, practice asking yourself what the proximate cause of that emotion was until it becomes a not-uncommon practice upon noticing a surprising emotion. To practice, you first need a way to notice emotions - either develop this, or ask someone else to cue you in whatever way the two of you choose, and precommit to ask yourself at their cue a) what your strongest emotion is currently, and b) what the largest components of that emotion's cause happen to be. I'm not sure what a good exercise for this might be, since it feels like you legitimately need an unanticipated emotion.

[-]MarkL14y10

Shinzen Young's "Do Nothing" Meditation:

http://www.youtube.com/watch?v=cZ6cdIaUZCA (Note that he is "neuroscience aware.")

http://www.shinzen.org/Retreat%20Reading/FiveWays.pdf

http://www.basicmindfulness.org/

(Also the discussion of "Willingness" in http://www.amazon.com/Acceptance-Commitment-Therapy-Second-Edition/dp/1609189620/)

The goal is to raise the baseline of equanimity and mindfulness. You naturally start allowing emotion and inner talk to thunder through you without letting it drive behavior. This is a prerequisite for ... (read more)

[-]lavalamp14y10

Whenever you find yourself about to express a belief, ask yourself if you really believe it it first. This works for me, but may not work for people not feeling curiosity or for people not already in the habit of being honest with themselves.

...often that thought causes me to not know what I believe.

Of course the hard part is figuring out a way to reliably put people in a situation where they will find themselves rationalizing. Social pressure would work on a lot of people, but probably not so well on LW types.

[-]HonoreDB14y10

This seems like it'll be easiest to teach and test if you can artificially create a preference for an objective fact. Can you offer actual prizes? Candy? Have you ever tried a point system and have people reacted well?

Assume you have a set of good prizes (maybe chocolate bars, or tickets good for 10 points) and a set of less-good prizes (Hershey's kisses, or tickets good for 1 point).

Choose a box: Have two actual boxes, labeled "TRUE" and "FALSE". Before the class comes in, the instructor writes a proposition on the blackboard, such a... (read more)

4JGWeissman14y

If the prize for correctly answering "true" is 10 times as good as the prize for correctly answering "false", then you really should be about 91% confident the correct answer is "false" before you give that answer.

0HonoreDB14y

Yup. The propositions need to be such that you can get more confident than that.

5JGWeissman14y

My point was that being biased to answer "true", even if "false" is more likely to be correct, is a rational strategy. This problem could be eliminated if the good effects of the correct answer being "true" were independant of getting the right answer. That is, if the correct answer is "true" you get 10 points, and if you answer correctly you get 1 point. That way, you want the answer to be "true", but it is not rational to let this have any effect on your answer.

0The Dao of Bayes14y

I assumed the point was to illustrate how, given the motivation (candy), your thinking DOES end up rationally biased towards saying "true". It gives you a clear example of motivated thinking (I want to answer "True" even if that's not the correct answer) and puts you inside the experience while still being entirely aware that it's motivated thinking.

0JGWeissman14y

That's not right. It is still the case that you want to answer "True" if the correct answer is "True", and you want to answer "False" if the correct answer is false. It's just that in the original formulation, the greater reward of correctly answer "True" means that you are willing to take a smaller chance of correctly answer "True" instead of a larger chance of correctly answering "False". Where in my modification, you want the correct answer to be "True", but if this actually influences whether you answer "True" or "False", you did something wrong.

0The Dao of Bayes14y

Sorry, that's what I meant by "I want to answer True even if it's not correct" :) Yours strikes me as teaching the skill "disentangle mixed incentives" - there's an incentive to be correct, and an incentive for the correct answer to be "TRUE". There's also the skill of recognizing which of these two incentives you have control over (the former only). While these are certainly USEFUL skills, I question whether this really helps people avoid Motivated Cognition. On an abstract level, it seems like they might be related, but I don't think it's the sort of exercise that would build an intuitive sense of "Oh, wait, I'm doing motivated cognition here!" I feel the original exercise, however, does a good job, because it puts the person in a position where they can SEE that they're doing motivated cognition. It lets them get a good look at what it FEELS like to be doing motivated cognition, what it looks like. Once they can recognize it, it's a LOT easier to fight it :)

[-][anonymous]10y00

I've reading about the CFAE exercise prizes. Now that there are some great tools I can learn, ay tips for mitigating the initial pain of going from any given irrational state to a more rational one? Any stories of how learning these rationality exercises have made an impact on some part of the quality of your lives?

[This comment is no longer endorsed by its author]Reply

[-]thomblake14y00

Exercise: Trick people into rationalizing

Engineer a situation where subjects will perform an action because you tricked them into doing it, and then ask them for reasons why they performed that action. Specifics here that actually work are beyond my pay-grade. But I'm reminded of an experiment where a subject was hypnotized and given an umbrella. After the hypnosis, the researcher asked "Why do you have an umbrella?" and the subject, initially looking a bit shocked at the umbrella, answered "I thought it was going to rain".

It would h... (read more)

[-]Benquo14y00

Think about a time when you were badly disappointed. Then think of any opportunities you had previously, to notice that things weren't going to turn out the way you'd expected. In particular, cases where you came across an argument or evidence that in hindsight should have changed your opinion, but you dismissed by a clever argument, or by not thinking about it.

Think about what you could have done in the time you had between your missed opportunity and the disappointment. Plans you could have changed to make the best of a bad situation. Mentally contrast the actual outcome with the outcome had you been forewarned, and allow yourself to be very annoyed at your past self for throwing away time.

[-]Incorrect14y00

Instructor writes a decision on the board and asks all participants to give factors to consider before making that decision. Participants are then asked as a group if each factor (for typical possible values of the factor) is 1. necessary and 2. sufficient to determine whether the decision should be taken.

[-]Incorrect14y00

I would ask Lucy if she would still eat the cake if the sugar industry were doing fine. More generally, I would ask if there is any action Lucy is taking that they would not take if the sugar industry were doing fine.

Ask the participants to come up with a belief that they feel is important to them but hasn't been necessary (though they may mistakenly think it sufficient) for making any meaningful decisions.

0Benquo14y

Perhaps Lucy would do well do ask: What if eating the cake somehow harmed the sugar industry? How would I feel about that?

[+]private_messaging14y-120

	It stays cold enough to require a sweater	It gets warm enough that no sweater is needed.
You believe you need a sweater	A warm walk in a toasty sweater.	Your walk is ruined forever by the need to carry an extra sweater.
You believe you don't need a sweater	You are cold! Cold cold cold! Why didn't you get a sweater?	Free and unencumbered, you stroll along as the warm Sun comes out overhead.

LESSWRONG
LW

LESSWRONG
LW

33

SotW: Avoid Motivated Cognition

33

33