Trivers' theory has been summed up by calling consciousness "the public relations agency of the brain". It consists of a group of thoughts selected because they paint the thinker in a positive light, and of speech motivated in harmony with those thoughts.

I found Modularity and the Social Mind: Are Psychologists Too Self-Ish? to be an excellent article relating to this. It also considerably helps question the concept of unified preferences.

And it also has plenty of other LW-related stuff and intriguing ideas packed into a very small space. In covers (and to me, clarifies) various ideas from modularity of mind, to the fact that having inconsistent beliefs need not cause dissonance, to our consciousness not being optimized for having true beliefs and being the PR firm instead of the president, and to the fact that any of our beliefs/behaviors that are not subjected to public scrutiny shouldn't be expected to become consistent. Very much recommended.

Abstract: A modular view of the mind implies that there is no unitary “self” and that the mind consists of a set of informationally encapsulated systems, many of which have functions associated with navigating an inherently ambiguous and competitive social world. It is proposed that there are a set of cognitive mechanisms—a social cognitive interface (SCI)—designed for strategic manipulation of others’ representations of one’s traits, abilities, and prospects. Although constrained by plausibility, these mechanisms are not necessarily designed to maximize accuracy or to maintain consistency with other encapsulated representational systems. The modular view provides a useful framework for talking about multiple phenomena previously discussed under the rubric of the self.

Some excerpts:

Taken together, the ideas that certain cognitive systems’ functions might not be designed to generate representations that are the best estimate of what is true along with the tolerance for mutually contradictory representations that modularity affords suggest a conclusion central to our overarching thesis. In particular, these two ideas imply that one cognitive subsystem can maintain a representation that is not the best possible estimate of what is true but can nonetheless be treated as “true” for generating inferences within the encapsulated subsystem. If a more accurate representation about the actual state of the world is represented elsewhere in the cognitive system, this presents no particular difficulty. Hence, there is no particular reason to believe that the mind is designed in such a way to maintain consistency among its various representational systems.

Summarizing, we have suggested the following. First, the mind consists of a collection of specialized systems designed by natural selection and furthermore, that individual systems are informationally encapsulated with respect to at least some types of information. Second, these systems have been selected by virtue of their functional consequences, not by virtue of their ability to represent what is true. Third, the encapsulation of modular systems entails that mutually contradictory representations can be simultaneously present in the same brain with no need for these representations to be reconciled or made mutually consistent.

[...]

We hypothesize that this is a primary function of the SCI: to maintain a store of representations of negotiable facts that can be used for persuasive purposes in one’s social world.3 For this reason, a crucial feature of the SCI is that it is not designed to maximize the accuracy of its representations, an idea consistent with the wealth of data on biases in cognitive processes (Greenwald, 1980; Riess, Rosenfeld, Melburg, & Tedeschi, 1981; Sedikides & Green, 2004). Instead, it is designed to maximize its effect in persuading others. As D. Krebs and Denton (1997) observed, “It is in our interest to induce others to overestimate our value” (p. 36). Humphrey and Dennett (1998) similarly concluded that “selves . . . exist primarily to handle social interactions” (p. 47).

There are, of course, limits to what others will believe. Because humans rely on socially communicated information, they have filtering systems to prevent being misled. Inaccuracy must be restrained. Thus, as a number of authors have pointed out, “Self-presentation is . . . the result of a tradeoff between favorability and plausibility” (Baumeister, 1999a, p. 8; see also D. Krebs & Denton, 1997; Schlenker, 1975; Schlenker & Leary, 1982a; Sperber, 2000a; Tice, Butler, Muraven, & Stillwell, 1995; Van Lange & Sedikides, 1998). The findings by Tice et al. (1995) that people are more modest in their selfpresentation to friends than to strangers is interesting in this regard, suggesting that others’ knowledge reigns in the positive features one can plausibly claim. This selection pressure might have led to an additional feature of the SCI: to maintain the appearance of consistency. This implies that one important design feature of the SCI is to maintain a store of representations that allow consistency in one’s speech and behavior that constitute the most favorable and defensible set of negotiable facts that can be used for persuasive purposes.

[...]

On our view, if the brain is construed as a government, the SCI, the entity that others in your social world talk to and the entity that talks back to others in your social world, is more like the press secretary than the president.5 The press secretary does not make major decisions or necessarily know how they were made, has access to only limited information from both below (sensory) and above (decision-making operations), and is in charge of spin. The press secretary will not always know what motivated various decisions and actions, although the press secretary is often called on to explain them.

[...]

Recall our claim that the existence of mechanisms designed to allow individuals to maintain and signal favorable and defensible representations of their characteristics would have led to selection pressures on perceivers to check for the accuracy of signalers’ communications. This would include systems designed to check communication against what else is known as well as systems to check communication for within-individual consistency (Sperber, 2000a). This in turn would have led to selection to maintain consistency in one’s communicative acts. If the SCI does not have access to the real causes of one’s own behavior (Freud, 1912/1999; Nisbett & Wilson, 1977), then this might induce the construction of a narrative to give causal explanations that are sensible, a task which must be accomplished without necessarily having the benefit of all potentially relevant information (Gazzaniga, 1998). Consistency is important with respect to the information other people possess—inconsistency entails minimal cost as long as the relevant facts cannot be assembled by others.

[...]

Although rarely pointed out, there are an extraordinarily large number of cases in which it is transparent that inconsistent representations are maintained with no effort to compensate in ways outlined in the initial theory (belief change, minimizing importance of discrepant representations, and so on). The most obvious cases are religious ideas, where beliefs thoroughly inconsistent with ontological commitments are deeply held. Indeed, it has been argued that it is precisely this discrepancy that causes these beliefs to be generated and transmitted (Boyer, 1994a, 1994b, 2001; Boyer & Ramble, 2001).

[...]

Returning to the first criterion, acts that are private and unlikely to become publicly known might similarly be relatively immune to the kind of reorganization implied by dissonance-related theories. This idea resonates with Tice’s (1992) suggestion that it is correct to “question whether internalization occurs reliably under private circumstances” (p. 447). Tice and Baumeister (2001) more recently suggested that “public behavior appears capable of changing the inner self” (p. 76), an idea that fits with Shrauger and Schoeneman’s (1979) finding that “individuals’ self-perceptions and their views of others’ perceptions of them are quite congruent” (p. 565), but that these same self-perceptions are not necessarily congruent with others’ actual perceptions. In other words, people try to maintain consistency with the way they think they are perceived (see also Baumeister, 1982; Baumeister & Cairns, 1992).

For example, although Aronson (1992, p. 305) emphasized preservation of one’s sense of self, Aronson, Fried, and Stone (1991) emphasized that it was “not practicing what they are preaching” (p. 1637) that can be expected to induce change. It is crucial to mark the distinction between “preaching” and the “self-concept.” Preaching is a social act, and predicting change as a function of this manipulation entails a commitment beyond preserving the self-concept (Aronson, 1992, 1999; see also Thibodeau & Aronson, 1992). An emphasis on hypocrisy (Aronson et al., 1991; Dickerson, Thibodeau, Aronson, & Miller, 1992; Fried & Aronson, 1995; Stone, Aronson, Crain, & Winslow, 1994; Stone, Wiegand, Cooper, & Aronson, 1997) that turns on inconsistencies in publicly known information (see, e.g., Stone et al., 1997), especially public advocacy (Fried & Aronson, 1995), implies the view that the preservation of concepts surrounding the self is insufficient to induce dissonance effects without the added social element.