Related: Cached Selves, The Neuroscience of Desire

You don't know your own mind.
    - Jonathan Swift, Polite Conversation

Researchers showed subjects two female faces for a few seconds and asked which face was more attractive. Researchers then placed the photos face down and handed subjects the face they had chosen, asking them to explain the motives behind their choice. But sometimes, researchers used a sleight-of-hand trick to switch the photos, showing viewers the face they had not chosen. Very few subjects noticed the face they were given was not the one they had chosen. Moreover, they happily explained why they preferred the face they had actually rejected, inventing reasons like "I like her smile" even though they had actually chosen the solemn-faced picture.¹

The idea that we lack good introspective access to our own desires - that we often have no idea what we want² - is a key lemma in naturalistic metaethics, so it seems worth a post to collect the science by which we know that.

Early warnings came from split-brain research, which identified an 'interpreter' in the left hemisphere that invents reasons for beliefs and actions. When the command 'walk' was flashed to split-brain subjects' right hemispheres, they got up from their chairs and start walking away. When asked why they suddenly started walking away, they replied (for example) that they got up because they wanted a Coke.³

The overjustification effect

Common sense suggests that we infer others' feelings from their appearance and actions, but we have a different, more direct route to our own feelings: direct perception or introspection.⁴ In contrast, self-perception theory⁵ suggests that our knowledge of ourselves is exactly like our knowledge of others.⁶ One famous result explained by self-perception theory is the overjustification effect.

In a famous 1973 study,

nursery school children drew pictures with a magic marker, a presumably intrinsically interesting activity, under one of three reward conditions. In the first condition the children expected to receive a reward (a fancy 'good player' award) for drawing, in the second they received the reward unexpectedly, and children in a third group received no reward. Only the expected reward produced a decrement in performance, during a later 'free play' period, as compared with the other two groups. [This] overjustification effect seemed to be due not to the reward itself but to the implication that the reward was the reason for the behavior. Only if the participants knew a reward was coming when they performed the behavior would it undermine their intrinsic interest in the task.⁷

It seems that subjects initially drew pictures because of intrinsic motivation in that activity, but the payment led them to unconsciously 'conclude' that their behavior did not represent their actual desires. Thus, they performed more poorly in the subsequent 'free play' period. This is known as the overjustification effect.

After dozens of similar studies, two meta-analyses confirmed that the overjustification effect occurs when (1) subjects are led to expect rewards before performing the behavior, (2) the rewards are tangible, and (3) the rewards are independent of the subjects’ level of performance.⁸

Implicit motivation

If we can be wrong about our own desires, then presumably many of our desires are activated unconsciously and operate unconsciously. Such implicit motivations have been amply confirmed.⁹

In one study, subjects were primed with achievement-related words ('strive', 'win', 'attain') during a word-finding task. During a second word-finding task, subjects were interrupted by an intercom announcement asking them to stop working. Those who had been primed with achievement-related words kept working more often than those who had not been so primed. Subjects were unable to identify the effect of this priming on their own motivation.¹⁰

This demonstrates that priming unconsciously affects the accessibility or strength of existing goals.¹¹ Do we unconsciously form goals, too?

We do, as shown by decades of research on operant conditioning. When a neutral potential goal is associated with a stimulus of positive affect, we acquire new goals, and we can be unaware that this has happened:

Watching someone smile while eating blueberry muffins may, for instance, link that activity to positive affect, which creates a goal representation. Indeed, such observational or social learning is thought to be a basic way in which infants learn which behavioral states are desired and which ones are not.¹²

Implications

This research is how we know about the hidden complexity of wishes, a key lemma in the fragility of human value. We don't know what many of our desires are, we don't know where they come from, and we can be wrong about our own motivations.

As such, we'd be unlikely to get what we really want if the world was re-engineered in accordance with a description of what we want that came from verbal introspective access to our motivations. Less naive proposals would involve probing the neuroscience of motivation at the algorithmic level.¹³

Notes

¹ Johansson et al. (2005).

² Several experiments have established that we infer rather than perceive the moment we decided to act: Rigoni et al (2010); Banks & Isham (2009, 2010); Moore & Haggard (2008); Sarrazin et al. (2008); Gomes (1998, 2002). But do not infer that conscious thoughts do not affect behavior. As on recent review put it: "The evidence for conscious causation of behavior is profound, extensive, adaptive, multifaceted, and empirically strong. However, conscious causation is often indirect and delayed, and it depends on interplay with unconscious processes. Consciousness seems especially useful for enabling behavior to be shaped by nonpresent factors and by social and cultural information, as well as for dealing with multiple competing options or impulses" (Baumeister et al. 2011). We can even be wrong about whether we intended to act at all: Lynn et al. (2010); Morsella et al (2010). If we don't have direct introspective access even to our decisions to act, why think we have introspective access to our desires?

³ Gazzaniga (1992), pp. 124-126.

⁴ But widespread findings of self-ignorance challenge this view. See, for example, Wilson (2004).

⁵ Zanna & Cooper (1974) seemed to have disproved self-perception theory in favor of cognitive dissonance theory, but Fazio et al (1977) showed that the two co-exist. This remains the modern view.

⁶ Laird (2007), p. 7.

⁷ Laird (2007), p. 126. The study described is Lepper et al. (1973).

⁸ Cameron & Pierce (1994); Tang & Hall (1995); Eisenberger & Cameron (1996).

⁹ Aarts & Dijksterhuis (2000); Bargh (1990); Bargh & Gollwitzer (1994); Chartrand & Bargh (1996, 2002); Fishbach et al. (2003); Fitzsimons & Bargh (2003); Glaser & Kihlstrom (2005); Gollwitzer et al. (2005); Hassin (2005); Shah (2003). For reviews, see Ferguson et al. (2007); Kruglanski & Kopetz (2008); Moskowitz et al. (2004). Unconscious motivations can even adapt to novel and changing circumstances: see Ferguson et al. (2007), pp. 155-157.

¹⁰ Bargh et al. (2001).

¹¹ Shah (2003); Aarts et al. (2004).

¹² Custers (2009).

¹³ Inferring desires from behavior alone probably won't work, either: Soraker & Brey (2007). Also: My thanks to Eliezer Yudkowsky for his feedback on a draft of this post.

References

Aarts & Dijksterhuis (2000). Habits as knowledge structures: Automaticity in goal-directed behavior. Journal of Personality and Social Psychology, 78: 53–63.

Aarts, Gollwitzer, & Hassin (2004). Goal contagion: Perceiving is for pursuing. Journal of Personality and Social Psychology, 87: 23–37.

Bargh (1990). Auto-motives: Preconscious determinants of social interaction. In Higgins & Sorrentino (eds.), Handbook of motivation and cognition: Foundations of social behavior (Vol. 2, pp. 93–130). Guilford.

Bargh & Gollwitzer (1994). Environmental control of goal-directed action: Automatic and strategic contingencies between situations and behavior. In Spaulding (ed.), Nebraska Symposium on Motivation (Vol. 41, pp. 71–124). University of Nebraska Press.

Bargh, Gollwitzer, Lee-Chai, Barndollar, & Troetschel (2001). The automated will: Nonconscious activation and pursuit of behavioral goals. Journal of Personality and Social Psychology, 81: 1014–1027.

Baumeister, Masicampo, & Vohs (2011). Do conscious thoughts cause behavior? Annual Review of Psychology, 62: 331-361.

Banks & Isham (2009). We infer rather than perceive the moment we decided to act. Psychological Science, 20: 17–21.

Banks & Isham (2010). Do we really know what we are doing? Implications of reported time of decision for theories of volition. In Sinnott-Armstrong & Nadel (eds.), Conscious Will and Responsibility: A Tribute to Benjamin Libet (pp. 47-60).

Cameron & Pierce (1994). Reinforcement, reward and intrinsic motivation: A meta-analysis. Review of Educational Research, 64: 363–423.

Chartrand & Bargh (1996). Automatic activation of impression formation and memorization goals: Nonconscious goal priming reproduces effects of explicit task instructions. Journal of Personality and Social Psychology, 71: 464–478.

Chartrand & Bargh (2002). Nonconscious motivations: Their activation, operation, and consequences. In Tesser & Stapel (eds.), Self and motivation: Emerging psychological perspectives (pp. 13–41). American Psychological Association.

Custers (2009). How does our unconscious know what we want? The role of affect in goal representations. In Moskowitz & Grant (eds.), The Psychology of Goals. Guilford.

Eisenberger & Cameron (1996). Detrimental effects of reward: Reality or myth? American Psychologist, 51: 1153–1166.

Fazio, Zanna, & Cooper (1977). Dissonance and self-perception: An integrative view of each theory's proper domain of application. Journal of Experimental Social Psychology, 13: 464-479.

Ferguson, Hassin, & Bargh (2007). Implicit Motivation. In Shah & Gardner (eds.), Handbook of Motivation Science (pp. 150-166). Guilford.

Fishbach, Friedman, & Kruglanski (2003). Leading us not unto temptation: Momentary allurements elicit automatic goal activation. Journal of Personality and Social Psychology, 84: 296–309.

Fitzsimons & Bargh (2003). Thinking of you: Nonconscious pursuit of interpersonal goals associated with relationship partners. Journal of Personality and Social Psychology, 84: 148–163.

Gazzaniga (1992). Nature's mind: The biological roots of thinking, emotion, sexuality, language, and intelligence. Basic Books.

Glaser & Kihlstrom (2005). Compensatory automaticity: Unconscious volition is not an oxymoron. In Hassin, Uleman, & Bargh (eds.), The new unconscious (pp. 171–195). Oxford University Press.

Gollwitzer, Bayer, & McCullouch (2005). The control of the unwanted. In Hassin, Uleman, & Bargh (eds.), The new unconscious (pp. 485–515). Oxford University Press.

Gomes (1998). The timing of conscious experience: a critical review and reinterpretation of Libet’s research. Consciousness and Cognition, 7: 559–595.

Gomes (2002). Problems in the timing of conscious experience. Consciousness and Cognition, 11: 191–97.

Hassin (2005). Non-conscious control and implicit working memory. In Hassin, Uleman, & Bargh (eds.), The new unconscious (pp. 196–224). Oxford University press.

Johansson, Hall, Silkstrom, & Olsson (2005). Failure to detect mismatches between intention and outcome in a simple decision task. Science, 310: 116-119.

Kruglanski & Kopetz (2008). The role of goal systems in self-regulation. In Morsella, Bargh, & Gollwitzer (eds.), Oxford Handbook of Human Action (pp. 350-369). Oxford University Press.

Laird (2007). Feelings: The Perception of Self. Oxford University Press.

Lepper, Green, & Nisbett (1973). Undermining children’s intrinsic interest with extrinsic rewards: A test of the 'overjustification' hypothesis. Journal of Personality and Social Psychology, 28: 129–137.

Lynn, Berger, Riddle, & Morsella (2010). Mind control? Creating illusory intentions through a phony brain–computer interface. Consciousness and Cognition, 19: 1007-1012.

Moore & Haggard (2008). Awareness of action: inference and prediction. Consciousness and Cognition, 17: 136–144.

Morsella, Berger, & Krieger (2010). Cognitive and neural components of the phenomenology of agency. Neurocase.

Moskowitz, Li, & Kirk (2004). The implicit volition model: On the preconscious regulation of temporarily adopted goals. In Zanna (ed.), Advances in experimental social psychology (Vol. 36, pp. 317–404). Academic Press.

Rigoni, Brass, & Sartori (2010). Post-action determinants of the reported time of conscious intentions. Frontiers in Human Neuroscience, 4: 38.

Sarrazin, Cleeremans, & Haggard (2008). How do we know what we are doing? Time, intention, and awareness of action. Consciousness and cognition, 17: 602–615.

Shah (2003). The motivational looking glass: How significant others implicitly affect goal appraisals. Journal of Personality and Social Psychology, 85: 424–439.

Soraker & Brey (2007). Ambient Intelligence and Problems with Inferring Desires from Behaviour. International Review of Information Ethics, 8: 7-12.

Tang & Hall (1995). The overjustification effect: A meta-analysis. Applied Cognitive Psychology, 9: 365–404.

Wilson (2004). Strangers to Ourselves: Discovering the Adaptive Unconscious. Belknap.

Zanna & Cooper (1974). Dissonance and the pill: An attribution approach to studying the arousal properties of dissonance. Journal of Personality and Social Psychology, 29: 703-709.