Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Wanting to Want

16 Post author: Alicorn 16 May 2009 03:08AM

In response to a request, I am going to do some basic unpacking of second-order desire, or "metawanting".  Basically, a second-order desire or metawant is a desire about a first-order desire.

Example 1: Suppose I am very sleepy, but I want to be alert.  My desire to be alert is first-order.  Suppose also that there is a can of Mountain Dew handy.  I know that Mountain Dew contains caffeine and that caffeine will make me alert.  However, I also know that I hate Mountain Dew1.  I do not want the Mountain Dew, because I know it is gross.  But it would be very convenient for me if I liked Mountain Dew: then I could drink it, and I could get the useful effects of the caffeine, and satisfy my desire for alertness.  So I have the following instrumental belief: wanting to drink that can of Mountain Dew would let me be alert.  Generally, barring other considerations, I want things that would get me other things I want - I want a job because I want money, I want money because I can use it to buy chocolate, I want chocolate because I can use it to produce pleasant taste sensations, and I just plain want pleasant taste sensations.  So, because alertness is something I want, and wanting Mountain Dew would let me get it, I want to want the Mountain Dew.

This example demonstrates a case of a second-order desire about a first-order desire that would be instrumentally useful.  But it's also possible to have second-order desires about first-order desires that one simply does or doesn't care to have.

Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict.  Obviously, as a heroin addict, she spends a lot of her time wanting heroin.  But this desire is upsetting to her.  She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.

One thing that is often said is that what first-order desires you "endorse" on the second level are the ones that are your most true self.  This seems like an appealing notion in Mimi's case; I would not want to say that at her heart she just wants heroin and that's an intrinsic, important part of her.  But it's not always the case that the second-order desire is the one we most want to identify with the person who has it:

Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual.  He has been brought up to believe that homosexuality is gross and wrong.  As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.

In this case, I think we're tempted to say that poor Larry is a gay guy who's had an alien second-order desire attached to him via his upbringing, not a natural homophobe whose first-order desires are insidiously eroding his real personality.

A less depressing example to round out the set:

Example 4: Suppose Olivia the Overcoming Bias Reader, whose very prescient mother predicted she would visit this site, is convinced on by Eliezer's arguments about one-boxing in Newcomb's Problem.  However, she's pretty sure that if Omega really turned up, boxes in hand, she would want to take both of them.  She thinks this reflects an irrationality of hers.  She wants to want to one-box.

 

1Carbonated beverages make my mouth hurt.  I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.

Comments (185)

Comment author: RobinHanson 16 May 2009 01:00:40PM 21 points [-]

I suspect most cases of "wanting to want" are better described as cases of internal conflict, where one part of us wishes that there weren't other parts of us with different conflicting wants.

Comment author: SoullessAutomaton 16 May 2009 01:12:55PM 9 points [-]

Particularly where one part is responsible for the "internal narrative" and the other is responsible for motivation and prioritization, because the latter usually wins out and the former complains loudest.

Comment author: nazgulnarsil 16 May 2009 03:29:58PM 8 points [-]

furthermore, the internal narrative has been carefully honed to be able to be disingenuous for signaling purposes.

Comment author: MichaelHoward 17 May 2009 03:23:18PM 3 points [-]

More to the point, the internal narrative part largely doesn't need to be disingenuous for signaling purposes, because it's kept in the dark about what the motivation and prioritization part is really up to.

Comment author: MichaelVassar 18 May 2009 11:05:45AM 4 points [-]

Agreed, but the parts in conflict may be of vastly different reflectivity. Some relevant parts may not have anything analogous to awareness of some other parts.

Comment author: Jack 16 May 2009 10:05:06PM 6 points [-]

Just so everyone is clear:

That is one way of describing cases where second order desires conflict with first order desires, perhaps. But one can want to to want X and want X... its just that Alicorn used only examples where the two conflict (and probably the distinction is best illustrated by looking at the conflicts). But right now I have both a first order desire not to use heroin and a second order desire to not use heroin. In fact, the vast majority of our desires are probably like this. So most cases of "wanting to want" are not cases of internal conflict but perhaps these cases can be described as instances of internal consistency.

Comment author: stcredzero 18 May 2009 03:17:56PM *  1 point [-]

In any case, I think Occam's Razor demands that we reject the notion of generalized second-order desires. We can leave that concept out entirely and explain everything as conflicts between first order desires and a generalized desire for consistency and/or resolution. Note that in all the examples, there are conflicting goals. In (1) it's the desire to stay awake vs. avoiding noxious stimuli. In (2) it's the desire to stay alive vs. cop another high. In (3) it's the desire to live up to his upbringing vs. follow his sexual urges.

I'm not even sure that a generalized desire for consistency and/or resolution would be a second-order desire. I think that the feeling of conflict over not being able to decide which speaker to buy is a lot like resolving conflicts between incompatible desires. The only difference is that choosing to buy a speaker is usually morally neutral, but there is societal pressure to choose one option as the "right" one in 2 and 3, and an imperative to preserve one's life in 1 and 2, so we are steered towards a particular outcome. It may well be only a trick of language that prompts us to say "I want to want X." I suspect that we could also say, "I want X and I want Y. I cannot have both, and I know I'm supposed to want X. I wish I wasn't conflicted." But that is much longer than, "I want to want X."

Comment author: freyley 16 May 2009 09:06:20PM 2 points [-]

"Better" in what way?

Do you mean better in that you think it's a more accurate view of the inside of your head?

Or better in that it's a more helpful metaphorical view of the situation that can be used to overcome the difficulties described?

I think the view of it as a conflict between different algorithms is useful, and it's the one that I start with, but I wonder whether different views of this problem might be helpful in developing more methods for overcoming it.

Comment author: JamesAndrix 17 May 2009 08:36:52PM 0 points [-]

The thing I'm getting from all this is: Any time you have two desires that turn out in the environment to be contradictory, you could also have a desire to change one of them (the 'lesser' one?)

But we don't always get this desire to want something different, I'm wondering if always should, never should, or if there is some clear rule.

Comment author: steven0461 16 May 2009 06:47:48PM *  0 points [-]

Seconded; more specifically it seems to me that if one does not want something but one wants to want it, it has to be the case that either:

  • there's one entity doing the not wanting, and some other entity that wants the one entity to stop doing the not wanting
  • one values wanting the thing for the sake of wanting the thing or for the sake of some result of wanting the thing other than getting the thing, not for the sake of getting the thing, and moreover one values wanting the thing so much that this outweighs the extra likelihood of getting the thing
  • one does not want to want the thing unconditionally, but one does want the thing to turn into a different thing that one does want (e.g. Alicorn's Mountain Dew example; there the thing one wants is to enjoy and want to drink Mountain Dew, which is not the same thing as wanting to want to drink Mountain Dew even though one does not enjoy it)
  • "wanting" here is some more informal human thing that isn't captured by rational decision theory (which is bad!)

Are those all the possibilities?

Comment author: PhilGoetz 17 May 2009 01:34:17AM *  9 points [-]

Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.

Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.

I'm really bothered by my inability to see how to distinguish between these two classes of meta-wants. I suppose you just punt it off to your moral system, or your expected-value computations.

Comment author: Ghatanathoah 24 October 2012 03:25:56AM 6 points [-]

Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn't really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn't the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn't wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.

Mimi the Heroin Addict, by contrast, probably doesn't want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin's negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.

Comment author: Vaniver 24 October 2012 03:28:45AM 3 points [-]

and mistakenly believes that homosexuality isn't the Right Thing

What reasoning process did you use to determine his belief was mistaken? When and where does Larry live? What are his other terminal goals?

Comment author: Ghatanathoah 24 October 2012 04:51:16AM 6 points [-]

In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.

The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results "Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal." I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of the same sex as the desirer.

Obviously if Larry lived in some intolerant hellhole of a country or time period it might be well to change his orientation to be bisexual or heterosexual so that he could satisfy his terminal goals of Sex and Romance without jeopardizing his terminal goals of Not Being Tortured and Killed. But that would be a second best the solution, the ideal solution would be to convince his fellows that their intolerance was unethical.

Comment author: Vaniver 24 October 2012 04:40:23PM 2 points [-]

PhilGoetz wrote:

I suppose you just punt it off to your moral system, or your expected-value computations.

I am having trouble seeing a significant difference between that and what you've described. Mimi's enabler could argue "human happiness is a Good Thing unless it seriously interferes with some other really important goal," and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the "serious interference" line. Similarly, the illegality of heroin and the illegality of homosexuality seem similarly comparable; perhaps Mimi should convince her fellows that their intolerance of her behavior is unethical.

Comment author: Ghatanathoah 25 October 2012 07:30:00AM *  20 points [-]

Let me try using an extended metaphor to explain my point: Remember Eliezer's essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?

Let's imagine a race of Pebblesorters that's p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.

Now let's suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn't a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.

The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn't a prime number. They begin to persecute 11-Pebble Favorers and imprison or kill them. They declare that Sortulon Prime, the mighty Pebblesorter God that sorts stars into gigantic prime-numbered constellations in the sky, is horribly offended that some Pebblesorters favor 11 pebble piles and will banish any 11-Pebble Favorers to P-Hell, where they will be forced to sort pebbles into heaps of 8 and 9 for all eternity.

Now let's take a look at an individual Pebblesorter named Larry the Closet 11-Pebble Favorer. He was raised by devout 13-Pebble Favorer parents and brought up to believe that 11 isn't a prime number. He has a second order desire to sort pebbles into prime-numbered heaps, and a first order desire to favor 11-pebble heaps. Larry is stricken by guilt that he wants to make 11-pebble heaps. He knows that 11 isn't a prime number, but still feels a strong first order desire to sort pebbles into heaps of 11. He wishes he didn't have that first order desire, since it obviously conflicts with his second order desire to sort pebbles into prime numbered heaps.

Except, of course, Larry is wrong. 11 is a prime number. His first and second order desires are not in conflict. He just mistakenly thinks they are because his parents raised him to think 11 wasn't a prime number.

Now let's make the metaphor explicit. Sorting pebbles into prime-numbered heaps represents Doing the Right Thing. Favoring 13-pebble heaps represents heterosexuality, favoring 11-pebble heaps represents homosexuality. Heterosexual sex and love and homosexual sex and love are both examples of The Right Thing. The people who think homosexuality is immoral are objectively mistaken about what is and isn't moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.

So the first and second order desires of Larry the Closet Homosexual and Larry the Closet 11-Pebble Favorer aren't really in conflict. They just think they are because their parents convinced them to believe in falsehoods.

I am having trouble seeing a significant difference between that and what you've described. Mimi's enabler could argue "human happiness is a Good Thing unless it seriously interferes with some other really important goal," and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the "serious interference" line.

Again, I assumed that Mimi was a psychologically normal human who had normal human second order desires, like having friends and family, being healthy, doing something important with her life, challenging herself, and so on. I assumed she didn't want to use heroin because doing so interfered with her achievement of these important second order desires.

I suppose Mimi could be a mindless hedonist whose second order desires are somehow mistaken about what she really wants, but those weren't the inferences I drew.

Mimi's enabler could argue "human happiness is a Good Thing unless it seriously interferes with some other really important goal,"

Again, recall my mention of a hypothetical Heroin 2.0 in my earlier comment. It seems to me that if Heroin 2.0 was suddenly invented, and Mimi still didn't want to use heroin, even though it no longer seriously interfered with her other important values, that she might be mistaken. Her second order desire might be a cached thought leftover from when she was addicted to Heroin 1.0 and she can safely reject it.

But I will maintain that if Larry and Mimi are fairly psychologically normal humans, that Mimi's second order desire to stop using heroin is an authentic and proper desire, because heroin use seriously interferes with the achievement of important goals and desires that normal humans (like Mimi, presumably) have. Larry's second order desire, by contrast, is mistaken, because it's based on the false belief that homosexuality is immoral. Homosexual desires do not interfere with important goals humans have. Rather, they are an important goal that humans have (love, sex, and romance), it's just that the objective of that goal is a bit unusual (same sex instead of opposite).

EDITED: To change some language that probably sounded too political and judgemental. The edits do not change the core thesis in any way.

Comment author: Eliezer_Yudkowsky 26 October 2012 04:28:09AM 8 points [-]

We should point people to this whenever they're like "What's special about Less Wrong?" and we can be like "Okay, first, guess how Less Wrong would discuss a reluctant Christian homosexual. Made the prediction? Good, now click this link."

Comment author: Epiphany 26 October 2012 07:21:48AM *  -2 points [-]

I'm surprised you regarded it so highly. The flaws I noticed are located in a response to Ghatanathoah's comment.

Comment author: Epiphany 26 October 2012 07:09:12AM *  6 points [-]

First, I would like to make one thing clear: I have absolutely nothing against homosexuals and in fact qualify as queer because my attractions transcend gender entirely. I call my orientation "sapiosexual" because it is minds that I am sexually attracted to, and good character, never mind the housing.

Stops at "pigheaded jerks"

downvotes

You know where this is going, oh yes, I am going right to fundamental attribution error and political mindkill.

The parents are deemed "pigheaded jerks" - a perception of their personality.

Larry the homosexual, convinced by the exact same reasoning, is given something subtly different - an attack on his behavior -- "he gullibly believed them" and you continue with "They (the Larrys) just think they are because their parents fed them a load of crap." attributing his belief to the situation that Larry is in.

Do you think Larry's grandparents didn't teach Larry's parents the same thing? And that Larry's great grandparents didn't teach it to Larry's grandparents?

This was a "good solid dig" at the other side.

Comment author: Ghatanathoah 26 October 2012 09:48:37AM 4 points [-]

You make an excellent point. I will edit my post to make it sound less political and judgemental.

Comment author: Epiphany 27 October 2012 12:31:28AM *  2 points [-]

I am charmed by your polite acknowledgement of the flaw and am happy to see that this has been updated. Thanks for letting me know that pointing it out was useful. :)

Comment author: MugaSofer 26 October 2012 08:33:00AM *  4 points [-]

I upvoted despite this. If you overlook that one problem, everything else is gold. That single flawed sentence does not effect the awesome of the other 14 paragraphs, as it does not contribute to the conclusion.

Comment author: Epiphany 27 October 2012 12:25:32AM *  -2 points [-]

My experience of it was more like:

"Oh, this is nice and organized... Still orderly... Still orderly... OHMYSPAGHETTIMONSTER I DID NOT JUST READ THAT!"

To me, it was a disappointment. Like if I were eating ice cream and then it fell to the ground.

If Eliezer is going to praise it like it's the epitome of what LessWrong should be, then it should be spotless. Do you agree?

Comment author: Vaniver 25 October 2012 10:59:51PM 4 points [-]

I think you're looking at this discussion from the wrong angle. The question is, "how do we differentiate first-order wants that trump second-order wants from second-order wants that trump first-order wants?" Here, the order only refers to the psychological location of the desire: to use Freudian terms, the first order desires originate in the id and the second order desires originate in the superego.

In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)

Some cases are easier than others- in the case of Sally, who wants to commit suicide but wants to not want to commit suicide, I would generally recommend methods of effective treatment for suicidal tendencies, not the alternative. But you should be able to recognize that the decision could be difficult, at least for some alteration of the parameters, and is the alteration is significant enough it could swing the other way.

There is also another factor which clouds the analysis, which is that the ego has to weigh the costs of altering, suppressing, or foregoing one of the desires. It could be that Larry has a twin brother, Harry, who is not homosexual, and that Harry is genuinely happier that Larry is, and that Larry would genuinely prefer being Harry to being himself; he's not mistaken about his second-order want.

However, the plan to be (or pretend to be) straight is much more costly and less likely to succeed than the plan to stop wanting to be straight, and that difference in costs might be high enough to determine the ego's decision. Again, it should be possible to imagine realistic cases in which the decision would swing the other way. (Related.)

It's also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it. (Note Alicorn's response. Given that homosexuality is immutable, then plans to change homosexuals are unlikely to succeed, and they might as well make the best of their situation. But I hope it's clear that, at its root, this is a statement about engineering reality, not moral principles- if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I'm sure you could think of some things to write in the "pro" column for "take the pill" and in the "con" column for "don't take the pill."

Why I said this is worth considering is that, as should be unsurprising, two wants conflict. Often, we don't expect the engineering reality to change. Male homosexuality is likely to be immutable for the lifetimes of the ones that are currently alive, and it's more emotionally satisfying to declare that homosexual desires don't conflict with important goals than reflect on the tradeoffs that homosexuals face that heterosexuals don't. Doing so, however, requires a sort of willful blindness, which may or may not be worth the reward gained by engaging in it.

Comment author: Ghatanathoah 26 October 2012 12:29:45AM 5 points [-]

if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I'm sure you could think of some things to write in the "pro" column for "take the pill" and in the "con" column for "don't take the pill."

I don't deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.

However, I would maintain that the specific reason given in Alicorn's orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.

It's also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it.

I find it illuminating to compare intuitions about homosexuality to intuitions about bisexuality. If homosexual relationships were really inferior to heterosexual ones in some important way then it would make sense to encourage bisexual people to avoid homosexual relationships and focus on heterosexual ones. This seems wrong to me however, if I was giving a bisexual person relationship advice I think the good thing to do would be advise them to focus on whoever is most compatible with them regardless of sex.

In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)

I think you are probably right, this is proper. I think I may feel biased in favor of second order desires because right now it seems like in my current life I have difficulty preventing my first order desires from overriding them. But if I think about it, it seems like I have many first order desires I cherish and would really prefer to avoid changing.

Comment author: JaySwartz 28 November 2012 09:33:44PM -1 points [-]

While the Freudian description is accurate relative to sources, I struggle to order them. I believe it is an accumulated weighting that makes one thought dominate another. We are indeed born with a great deal of innate behavioral weighting. As we learn, we strengthen some paths and create new paths for new concepts. The original behaviors (fight or flight, etc.) remain.

Based on this known process, I conjecture that experiences have an effect on the weighting of concepts. This weighting sub-utility is a determining factor in how much impact a concept has on our actions. When we discover fire burns our skin, we don't need to repeat the experience very often to weigh fire heavily as something we don't want touching our skin.

If we constantly hear, "blonde people are dumb," each repetition increases the weight of this concept. Upon encountering an intelligent blond named Sandy, the weighting of the concept is decreased and we create a new pattern for "Sandy is intelligent" that attaches to "Sandy is a person" and "Sandy is blonde." If we encounter Sandy frequently, or observe many intelligent blonde people, the weighting of the "blonde people are dumb" concept is continually reduced.

Coincidentally, I believe this is the motivation behind why religious leaders urge their followers to attend services regularly, even if subconsciously. The service maintains or increases weighting on the set of religious concepts, as well as related concepts such as peer pressure, offsetting any weighting loss between services. The depth of conviction to a religion can potentially be correlated with frequency of religious events. But I digress.

Eventually, the impact of the concept "blonde people are dumb" on decisions becomes insignificant. During this time, each encounter strengthens the Sandy pattern or creates new patterns for blondes. At some level of weighting for the "intelligent" and "blonde" concepts associated to people, our brain economizes by creating a "blond people are intelligent" concept. Variations of this basic model is generally how beliefs are created and the weights of beliefs are adjusted.

As with fire, we are extremely averse to incongruity. We have a fundamental drive to integrate our experiences into a cohesive continuum. Something akin to adrenaline is released when we encounter incongruity, driving us to find a way to resolve the conflicting concepts. If we can't find a factual explanation, we rationalize one in order to return to balanced thoughts.

When we make a choice of something over other things, we begin to consider the most heavily weighted concepts that are invoked based on the given situation. We work down the weighting until we reach a point where a single concept outweighs all other competing concepts by an acceptable amount.

In some situations, we don't have to make many comparisons due to the invocation of very heavily weighted concepts, such as when a car is speeding towards us while we're standing in the roadway. In other situations, we make numerous comparisons that yield no clear dominant concept and can only make a decision after expanding our choice of concepts.

This model is consistent with human behavior. It helps to explain why people do what they do. It is important to realize that this model applies no division of concepts into classes. It uses a fluid ordering system. It has transient terminal goals based on perceived situational considerations. Most importantly, it bounds the recursion requirements. As the situation changes, the set of applicable concepts to consider changes, resetting the core algorithm.

Comment author: NancyLebovitz 28 November 2012 10:45:14PM 2 points [-]

From what I've heard, the typical response to believing that blond people are dumb and observing that blond Sandy is intelligent is to believe that Sandy is an exception, but blond people are dumb.

Most people are very attached to their generalizations.

Comment author: JaySwartz 29 November 2012 03:52:23PM -1 points [-]

Quite right about attachment. It may take quite a few exceptions before it is no longer an exception. Particularly if the original concept is regularly reinforced by peers or other sources. I would expect exceptions to get a bit more weight because they are novel, but no so much as to offset higher levels of reinforcement.

Comment author: CCC 26 October 2012 07:38:56AM 2 points [-]

Your example does an exemplery job of explaining your viewpoint on Larry's situation. To explain the presumed viewpoint of Larry's parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9.

The people who think homosexuality is immoral are objectively mistaken about what is and isn't moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.

How do you define objective morality? I've heard of several possible definitions, most of which conflict with each other, so I'm a little curious as to which one you've selected.

Comment author: Ghatanathoah 26 October 2012 09:37:17AM *  2 points [-]

To explain the presumed viewpoint of Larry's parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9.

I'm not sure I understand you, do you mean that a more precise description of Larry's parent's viewpoint is that the Pebblesorter versions of them think 11 and 9 are the same numbers? Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument.

How do you define objective morality?

I define morality as being a catch-all term to describe what are commonly referred to as the "good things in life," love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things. In other words, "good" and "right" are synonyms. Morality is objective because we can objectively determine whether people are happy, being treated fairly, getting what they want out of life, etc. In Larry's case having a relationship with Ted-the-Next-Door neighbor would be the morally right thing to do because it would increase the amount of love, happiness, people-getting what they want, etc. in the world.

I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism. That is, they believe that moral knowledge is intrinsically motivating, simply knowing something is right motivates someone to do it. They then conclude that since intrinsically motivating knowledge doesn't seem to exist, that morality must be subjective.

I am a motivational externalist, so I do not buy this argument. I believe that that people are motivated to act morally by our conscience and moral emotions (i.e. compassion, sympathy). If someone has no motivation to act to increase the "good things in life" that doesn't mean morality is subjective, that simply means that they are a bad person. People who lack moral emotions exist in real life, and they seem to lack any desire to act morally at all, unless you threaten to punish them if they don't.

The idea of intrinsically motivating knowledge is pretty scary if you think about it. What if it motivated you to kill people? Or what if it made you worship Darkseid? The Anti-Life equation from Final Crisis works pretty much exactly the way motivational internalists think moral knowledge does, except that instead of motivating people to care about others and treat people well, it instead motivates them to serve evil pagan gods from outer space.

Comment author: CCC 26 October 2012 10:59:28AM 4 points [-]

Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument.

Yes, exactly. Larry's parents' do not believe that they are mistaken, and are not easily proved mistaken.

I define morality as being a catch-all term to describe what are commonly referred to as the "good things in life," love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things.

That's a good definition, and it avoids most of the obvious traps. A bit vague, though. Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative... except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I'd still think that leaving the child locked away in the dark is a wrong thing. (This can also lead to Pascal's Mugging, as an edge case)

I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism.

In my case, it's because every attempt I've seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche? There are some bad things in the dark corners of the human psyche. Take whatever action is most likely to transform the world into a paradise? Doesn't usually work, because we don't know enough to always select the correct actions. Do unto others as you would have them do unto you? That's a very nice one - but not if Bob the Masochist tries to apply it.

Of course, subjective morality is no better - and is often worse (mainly because a society in general can reap certain benefits from a shared idea of morality).

What does seem to work is to pick a society whose inhabitants seem happy and fulfilled, and trying to use whatever rules they use. The trouble with that is that it's kludgy, uncertain, and could often do with improvement (though it's been improved often enough in human history that many - not all, but many - obvious 'improvements' aren't in practice).

Comment author: Nornagest 27 October 2012 12:46:21AM *  5 points [-]

Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative... except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I'd still think that leaving the child locked away in the dark is a wrong thing.

Aside from its obvious artificiality, and despite the fact that all our instincts cry out against it, it's not at all clear to me that there are any really good reasons to reject the Omelasian solution. This is of course a fantastically controversial position (just look at the response to Torture vs. Dust Specks, which might be viewed as an updated and reframed version of the central notion of The Ones Who Walk Away From Omelas), but it nonetheless seems to be a more or less straightforward consequence of most versions of consequential ethics.

As a matter of fact, I'm inclined to view Omelas as something between an intuition pump and a full-blown cognitive exploit: a scenario designed to leverage our ethical heuristics (which are well-adapted to small-scale social groups, but rather less well adapted to exotic large-scale social engineering) in order to discredit a viewpoint which should rightfully stand or fall on pragmatic grounds. A tortured child is something that hardly anyone can be expected to think straight through, and trotting one out in full knowledge of this fact in order to make a point upsets me.

Comment author: Ghatanathoah 26 October 2012 12:39:00PM 2 points [-]

A bit vague, though.

That's true, but I think that human values are so complex that any attempt to compress morality into one sentence is pretty much obligated to be vague.

Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative... except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests.

One rather obvious rejoinder is that there are currently hundreds, if not thousands of children who are in the same state as the unfortunate Omelasian right now in real life, so reducing the number to just one child would be a huge improvement. But you are right that even one seems too many.

A more robust possibility might be to add "equality" to the list of the "good things in life." If you do that then Omelas might be morally suboptimal because the vast inequality between the child and the rest of the inhabitants might overwhelm the achievement of the other positive values. Now, valuing equality for its own sake might add other problems, but these could probably be avoided if you were sufficiently precise and rigorous in defining equality.

In my case, it's because every attempt I've seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche?

I think the best explanation I've seem is something like the metaethics Eliezer espouses, which is (if I understand them correctly), that morality is a series of internally consistent concepts related to achieving what I called "the goods things in life," and that human beings (those who are not sociopaths anyway) care a lot about these concepts of wellbeing and want to follow and fulfill them.

In other words, morality is like mathematics in some ways, it generates consistent answers(on the topic of people's wellbeing) that are objectively correct. But it is not like the Anti-Life Equation because it is not intrinsically motivating. Humans care about morality because of our consciences and our positive emotions, not because it is universally compelling.

To put it another way, I think that if you were to give a superintelligent paperclipper a detailed description of human moral concepts and offered to help it make some more paperclips if it elucidated these concepts for you, that it would probably generate a lot of morally correct answers. It would feel no motivation to obey these answers of course, since it doesn't care about morality, it cares about making paperclips.

This is a little like morality being "embedded in the human psyche" in the sense that the desire to care about morality is certainly embedded in their somewhere (probably in the part we label "conscience"). But it is also objective in the sense that moral concepts are internally consistent independent of the desires of the mind. To use the Pebblesorter metaphor again, caring about sorting pebbles into prime numbered heaps is "embedded in the Pebblesorter psyche," but which numbers are prime is objective.

There are some bad things in the dark corners of the human psyche.

That's certainly true, but that simply means that humans are capably of caring about other things besides morality, and these other things that people sometimes care about can be pretty bad. This obviously makes moral reasoning a lot harder, since it's possible that one of your darker urges might be masquerading as a moral judgement. But that just means that moral reasoning is really hard to do, it doesn't mean that it's wrong in principle.

Comment author: nshepperd 26 October 2012 11:24:25AM 1 point [-]

What does seem to work is to pick a society whose inhabitants seem happy and fulfilled, and trying to use whatever rules they use.

If you're going to do that, why not just directly use happiness and fulfillment?

Comment author: Peterdjones 26 October 2012 07:50:36AM 0 points [-]

Define it, or defend it? There are a lot of defences, but not so much definitions.

Comment author: Jayson_Virissimo 26 October 2012 07:36:40AM 1 point [-]

I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters' ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?

Comment author: wedrifid 26 October 2012 10:00:45AM *  2 points [-]

I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters' ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?

Preferring other people like Larry to be homosexual is adaptive for me. And it is the judgement by others (and the implicit avoidance of that through shame) that we are considering here. That said:

I think the metaphor misses something important here

Absolutely, and the entire line of reasoning relies on conveying the speaker's own morality ("it is second-order 'right' to be homosexual") on others without making it explicit.

Comment author: MugaSofer 26 October 2012 08:23:37AM *  0 points [-]

The same reason sorting pebbles into correct heaps was adaptive in the first place.

EDIT: Wait, does it matter that homosexuality is probably not adaptive?

Comment author: Jayson_Virissimo 26 October 2012 08:48:37AM 0 points [-]

Wait, does it matter that homosexuality is probably not adaptive?

That was the point of my comment. There is a large disanalogy between heterosexuality and 13-pebble heap preference (namely, the first highly adaptive, but the second has no apparent reason to be). Although, I'm not sure if that is enough to break the metaphor.

Comment author: MugaSofer 26 October 2012 09:01:12AM 1 point [-]

There are many properties homosexuality has but 11-pebble heap preference don't, and vice versa. Why is evolutionary maladaptiveness worth pointing out, is my question.

Comment author: MugaSofer 26 October 2012 09:56:10AM 0 points [-]

Incidentally, it's easier to sort pebbles into heaps of 11. The original pebblesorters valued larger heaps, but had a harder time determining their correctness.

Comment author: orthonormal 18 May 2009 07:03:18PM 2 points [-]

I suspect we're doing some extrapolation here in order to distinguish these cases. I expect that if Mimi knew more about herself and the world, and thought more clearly, she would still want to not want heroin; while I expect that if Larry knew more about himself and the world, and thought more clearly, he would be likely to reject the system of belief that causes him to think homosexuality immoral.

Comment author: mitechka 18 May 2009 07:28:04PM 0 points [-]

Alternatively, after sobering up, Mimi might decide that experiencing heroine high makes her life so much more fulfilling, that the much shortened life expectancy of a heroine addict doesn't seems to be a fair price to pay for it.

As usual it is all up to personal definition of utility.

Comment author: Alicorn 17 May 2009 02:48:31AM *  -1 points [-]

I think it probably has something to do with the fact that Mimi (probably) wasn't born addicted to heroin (and even if she was, we can point to the behavior that caused it), whereas the consensus seems to be that homosexuality is innate.

Comment author: PhilGoetz 17 May 2009 04:07:35AM 2 points [-]

There must be more to it than that. If Larry were innately born attracted to children rather than to men, we probably wouldn't say it was okay.

Comment author: Alicorn 17 May 2009 04:19:20AM 0 points [-]

It's not about whether it's okay, it's about whether it's "part of who he is" or an alien intrusion.

Comment author: RichardKennaway 19 May 2009 10:46:20AM 3 points [-]

It's not about whether it's okay, it's about whether it's "part of who he is" or an alien intrusion.

That doesn't solve PhilGoetz's example though. And in the original version of Larry, his parents might very well say that his revulsion at homosexual acts is "who he is" and his sexual feelings the "alien intrusion". Are these concepts anything but a way of making disguised moral judgements? Is "who someone really is" just "who I would prefer them to be"?

Then again, another attitude to Larry is that his sexual feelings are who he really is, but that resisting them is a cross he has to bear. (I believe this is the Roman Catholic view.) So I don't think the concept of authenticity solves these problems.

Comment author: newerspeak 17 May 2009 08:52:04AM *  3 points [-]

... "part of who he is" or an alien intrusion.

Okay.

I'm Paul Erdos. I've been taking amphetamine and ritalin for 20-odd years to enhance my cognitive performance. In general I want to want these drugs, because they help me do good, important and enjoyable work, which is impossible for me without them.

I can stop wanting these drugs when I want to, like when my friend bet me $500 that I couldn't. I wanted to win that bet, so I wanted not to want the drugs, so I stopped wanting them. Was that my only motivation?

Also, I don't want others to want to want amphetamines just because I want to want amphetamines.

A while ago I took Euler's place as the most prolific mathematician of all time.

Comment author: ABranco 13 October 2009 05:36:57PM 1 point [-]

Paul Erdös did it regularly, yes. Successfully, it seems — but I wonder about the costs. Does anyone have consistent data on that?

Picking only Erdös' case, would, I'm afraid, be a case of both survivorship bias and hasty generalization.

Comment author: MichaelBishop 17 May 2009 05:47:34PM -1 points [-]

Contra Cyan & Alicorn, I am inclined to go with PhilGoetz and "punt it off to your moral system, or your expected-value computations."

Trying to change your homosexual desires will probably fail and create a lot of collatoral damage. I would guess that trying to change your desire for heroin is somewhat more likely to succeed, though I'm willing to consider the argument that heroin addicts should accept their addiction but attempt to minimize its harmful side effects.

Comment author: Cyan 17 May 2009 02:51:29AM *  -1 points [-]

I think the distinction is that we think of Mimi as wishing to revoke a decision made of her own free will; not so with Larry.

Comment author: AnlamK 16 May 2009 09:59:54AM *  4 points [-]

Harry Frankfurt, who came up with the original idea, did a much better job in explaining in my opinion. (Why are you not referring to his paper?)

Here is the link for the curious: http://www.usfca.edu/philosophy/pdf%20files/Freedom%20of%20the%20Will%20and%20the%20Concept%20of%20a%20Person.pdf

Comment author: Alicorn 16 May 2009 03:10:38PM *  1 point [-]

I probably should have mentioned Frankfurt's work, but I was being petty and declined to do so because he irritates me by calling second-order desire a criterion for personhood. Moreover, I wasn't trying to get into the notion of "will" or what second-order desire is for; I just wanted to provide a summary and some examples because someone had asked about it, and if the post is well-received I'll follow up with more complicated stuff.

Comment author: thomblake 19 May 2009 04:16:16PM 1 point [-]

Still, at least a hat-tip is obviously warranted.

I probably should have mentioned Frankfurt's work, but I was being petty and declined to do so because he irritates me by calling second-order desire a criterion for personhood.

Hardly an excuse for academic dishonesty. Okay, this forum is hardly 'academic', but the point stands.

Comment author: Jordan 16 May 2009 08:15:13AM *  2 points [-]

It's not always so easy to say which desire is actually first order and which is second order.

For instance, example 3 could be inverted:

Larry was brought up to believe God hates homosexuality. Because of this he experiences genuine disgust when he thinks about homosexual acts, and so desires not to perform them or even think about them (first order). However, he really likes his friend Ted and sometimes wished God wasn't such a dick (second order).

There's likely even a third order desire: Larry was brought up to be a good Christian, and desperately wishes he didn't wish God was anything other than He is (third order).

I imagine our desires are less like a logical hierarchy and more like a food chain. On any given day Larry's libido could be the biggest fish in the sea.

Comment author: Psychohistorian 16 May 2009 10:52:59AM *  1 point [-]

wished God wasn't such a dick (second order).

This is not second order. It's just D(God approved of homosexuality). If he himself were God, then it would probably be second order, but just wanting the rules to be different is first-order. Similarly,

Larry was brought up to be a good Christian, and desperately wishes he didn't wish God was anything other than He is (third order).

is not third order. Third order gets weird. Third order would be that Mimi wants heroin, and in fact, wants to want heroin, (if you offered to magically make her not like heroin, she'd emphatically decline), but on top of that, she wants to want to not want to use heroin. Maybe she doesn't have a problem with it, but her friends do. Maybe she would be well served if she could express an honest desire to quit while not actually quitting. Third-order desires get pretty confusing, though I may have just explained this one poorly.

Furthermore

I imagine our desires are less like a logical hierarchy and more like a food chain. On any given day Larry's libido could be the biggest fish in the sea.

is not exactly in line with the post. I don't think there's any claim that these things are a logical hierarchy. One can have extremely weak first order desires and extremely strong second order desires, though the latter will tend to consolidate into changed first order desires if they are strong enough.

Second order desires do often stem from conflicting first order desires, but what determines the order is the object of desire (if it itself is about a desire, it's 2nd order, and so on), not its magnitude.

Comment author: Jordan 16 May 2009 04:36:22PM *  1 point [-]

I can see that I worded things in a misleading fashion. The main point is just that first orders desires and second order desires referring to them can often switch places. Rephrasing:

Larry wants to be straight... (first order)

...but wants not to want to be straight so he can be with his friend Ted. (second order)

I imagine our desires are less like a logical hierarchy and more like a food chain.

is not exactly in line with the post. I don't think there's any claim that these things are a logical hierarchy.

By logical hierarchy I just mean the notion there is some well ordering of terms here. Each desire can be the first order desire to the other, forming a loop rather than a tier.

Comment author: Psychohistorian 16 May 2009 07:48:17PM 2 points [-]

First , these two statements are 2nd order and 3rd order respectively (taboo "straight" and you get -very roughly- "Larry wants to want to have sex with women" ). Second, they are not representations of the same thing, since they point in opposite directions. Thus, they don't seem to support the claim that first and second order desires can switch places.

More importantly, you can't just infer n-th order desires from n-m, n>m>0 order desires.

"I want chocolate," does not imply: "I do not want to want to want to not want to want to not want to not want to want to want to not want to not want chocolate," even though the two happen to point in the same direction. The first one is true, the second one is almost certainly false, since I really don't think that hard about chocolate.

Higher-order desires get really, really convoluted if you use imprecise language, and they're not exactly simple to begin with.

I am starting to agree with other posters that the whole construction may not map reality too accurately, but if you actually use precise language, n-th order desires are distinct and meaningful. Without precise language, you're just showing it's possible to say the same thing in more than one way, which is true, but not insightful.

Comment author: Jordan 16 May 2009 11:34:53PM 0 points [-]

More importantly, you can't just infer n-th order desires from n-m, n>m>0 order desires.

I am starting to agree with other posters that the whole construction may not map reality too accurately

I agree. Generally people don't have very high order desires. Discounting the accuracy/usefulness of the notion of ordered desires was the entire thrust of my original comment.

First , these two statements are 2nd order and 3rd order respectively

This is still a matter of interpretation based on wording. In my original phrasing it's more apparent that the desire is first order. My set up was that Larry "experiences genuine disgust when he thinks about homosexual acts", the intent being that his reaction to homosexuality is involuntary, and his desire is simply to avoid that unpleasant reaction. This is first order.

I'm not claiming the order can always be reversed. I was just giving a particular construction where it could.

Comment author: pjeby 16 May 2009 05:05:30PM 5 points [-]

The main point is just that first orders desires and second order desires referring to them can often switch places.

Actually, it's simpler to treat all desires as independent motivations. Whether they are "first-order" or "second-order" is a function of which is the organism's current goal.

When an addict feels bad, their current goal is to not feel bad - so they indulge. Anything that would stop them from doing so -- including any desire to reform -- is now opposed, whether the thing being opposed is an external event or an internal desire to quit.

Conversely, when the addict is satisfied, their current goal may be to feel better about themselves, at which point whatever obstacles to that goal become relevant, whether external events or internal desires.

IOW, it's not the case that desires are ever "first order" or "second order" in and of themselves. So-called "second order" desires are merely subgoals that arise as a side-effect of a conflict between an active goal and another desire that opposes the goal in some way.

And it's important to understand their contextual nature. A subgoal that's consistently reinforced can become a seemingly-independent desire via the standard "cached thought" or "promoted subgoal" mechanism, but this doesn't always occur. And until it occurs, the second-order desires are just temporary subgoals. See for example, Alicorn's example 1: the goal of liking Mountain Dew (or not disliking it) is strictly in the context of the alertness goal. If the alertness goal were satisfied in some other way, the subgoal would no longer be active.

Comment author: MichaelVassar 18 May 2009 11:11:44AM 0 points [-]

I'm pretty sure that you are pointing to a correct proposition here but have overstated your case. At the very least, separate from goals its possible to build non-desire habits which desires can affirm or conflict with. More importantly, you are missing the point that second order desires can be about first order desires without the first order desires, even when active, being about the second order desires.

Comment author: pjeby 18 May 2009 02:20:22PM 0 points [-]

More importantly, you are missing the point that second order desires can be about first order desires without the first order desires, even when active, being about the second order desires.

How so? That is, how am I missing that point? I'm simply saying that second-order desire can only arise as a subgoal of some other desire, even if that other desire is to simply have a certain social image. This doesn't imply any sort of symmetry being required, so I'm not clear on why you think I said it does. (In fact, my reference to example 1 describes an asymmetric conflict case.)

Comment author: stcredzero 18 May 2009 04:17:16PM 0 points [-]

Asymmetry between 2nd order and 1st order desires can be explained easily if 2nd order desires didn't really exist, or if only one specific 2nd order desire existed, namely a desire for resolution. "I want to want X" then just becomes some person's rationalization about their conflicted situation. I find this idea attractive, because a desire for resolution seems a natural thing for a conscious being to have.

Comment author: pjeby 18 May 2009 04:48:27PM 0 points [-]

I find this idea attractive, because a desire for resolution seems a natural thing for a conscious being to have.

But such a thing isn't intrinsic. People routinely do things that are in conflict, without ever resolving the conflict. If anything, we have a drive to appear consistent to other people -- a more evolutionarily-relevant drive than a desire to actually be consistent.

Meanwhile, second-order desires are just subgoals, like "walk across the room" is a subgoal of "get a glass of water". We experience wanting to (not) want something because it supports some other goal -- whether the other goal is something we want to admit to or not.

But that other goal is never really "get some resolution" -- that's just a verbal explanation that deflects attention from whatever the real goal is. (Because without some conflicting goal being present, there would be nothing to "resolve"!)

Comment author: stcredzero 18 May 2009 05:59:21PM *  0 points [-]

I disagree. Conflict resolution is intrinsic, however most people resolve many of their conflicts in irrational ways, including distraction. Come to think of it, a desire/goal-conflict resolution urge explains procrastination quite handily.

Let's not confuse "a desire to resolve conflicting urges" with "a desire to be rationally self-consistent." These are two different things. Everyone will have the former. Some will be able to cultivate the latter. A drive to appear consistent to others is yet a third thing.

I suspect that our drive for "getting resolution" is much like our preference for clearly annunciated speech, sunny vistas, and uncluttered rooms. We are driven to optimize our perception, and this drive is expressed as aesthetic desire. Our sense of the aesthetic even extends to internal perception of our ideas -- there is an attraction to elegant ideas. Religions often exploit this. For example, Islam is said to be popular in parts of the world because it presents itself as straightforward.

I think you're right that "getting resolution" is not a goal. It is more like a drive. Much like our desire to see all of what we are observing often results in our craning our neck. Like other drives it can result in goals. I also like your subgoal formulation. I would posit that the urge towards "clarity" is what drives it. But remember, just because one can imagine some scenario and take it as a desired goal, doesn't mean that the situation is sensible. I think wanting to want X is along the same lines as wanting to hear the sound of one hand clapping.

So, in the Mountain Dew example, the subject wants to stay awake and the subject also wants to avoid the unpleasant stimuli of Mountain Dew. To resolve this goal conflict, they formulate the subgoal, "I want to want Mountain Dew," which is a condition where there is no conflict. I note, however, that the subject wouldn't mind drinking chilled but flat Mountain Dew if it were readily available. Most likely they would immediately want to drink it. I posit that they always wanted to drink the Mountain Dew, but that they had a conflicting goal (that of avoiding carbonation), and were distracted by a poorly formulated subgoal.

Comment author: pjeby 18 May 2009 08:08:08PM 0 points [-]

Let's not confuse "a desire to resolve conflicting urges" with "a desire to be rationally self-consistent." These are two different things. Everyone will have the former. Some will be able to cultivate the latter. A drive to appear consistent to others is yet a third thing.

My point is that "a desire to resolve conflicting urges" is an unnecessary hypothesis. Conflict resolution is an emergent property of goal-seeking, not an independent goal or desire of itself, nor even a component of goal-seeking.

If you have a goal to get a soda from the fridge, and therefore a subgoal of walking across the room, but there is something in your way, then you will desire to go around it. To posit even a "drive" to "get resolution" is adding unnecessary entities to the equation.

Now, if you said that we experience conflict as painful, and desire to avoid it, I'd agree with you. However, experiencing the pain of conflict does not consistently motivate people to resolve the conflict. In fact, it frequently motivates people to avoid the subject entirely, so as to remove awareness of the conflict!

That's why I believe that talking about "conflict resolution as intrinsic" or an urge to "get resolution" is both unnecessary and erroneous: people DO experience negative reinforcement from conflict, but this is not the same thing as a desire for resolution. In humans (as in all animals that I know of), a drive to avoid one thing does not produce the same results as a drive to approach its opposite (nor vice versa).

Comment author: Alicorn 18 May 2009 06:05:24PM *  0 points [-]

This is kind of tangential to your actual statement, but I've never found an originally carbonated beverage that was flat enough to be drinkable. I'd think it had more to do with the flavorings used in sodas if I didn't have the same problem with seltzer water and sparkling juices.

Comment author: Vladimir_Nesov 18 May 2009 10:45:27PM 0 points [-]

Of two conflicting desires, we call second-order the one we don't expect to go away, and as such more invariant, part of self, even if not ever in control.

Most second-order desires are not about first-order desires, they are about the same thing as the first-order desire. For the second-order desire, modifying the first-order desire is instrumental, not terminal, and the same applies in the other direction. The differences I see come from first-order desire being actually in control, and being stupid enough not to work on eliminating the second-order desire.

Comment author: conchis 16 May 2009 11:33:14AM 0 points [-]

Third order would be that Mimi wants heroin, and in fact, wants to want heroin,

Isn't that just second order? Third order would be wanting to want to want heroin.

Comment author: Psychohistorian 16 May 2009 12:16:58PM 0 points [-]

It included the next sentence (wanting to want to not want); edited to make it less ambiguous.

Comment author: conchis 16 May 2009 12:48:31PM 0 points [-]

Ah. Sorry!

Comment author: pjeby 16 May 2009 02:20:21PM 1 point [-]

There's a simpler model for all of these examples -- you're describing conflicts between an "away-from" motivation and a "towards" motivation. These systems are semi-independent, via affective asynchrony. The second-order want is then arising as a subgoal of the currently-active goal (be alert, etc.).

I guess what I'm trying to say here is that there really aren't "second order wants" in the system itself; they're just an emergent property of a system with subgoals that explicitly models itself as an agent, especially if it also has goals about "what kind of person" it is desirable or undesirable to be.

It's likely that examples 2 and 4 would both be based in self-image goals, as well as the more obvious example in 3. Clearly, non-self-image cases like #1 exist too, so it's not strictly about such things, but self-image goals (whether "toward" or "away from") are the most common source of lasting and emotionally-distressing conflict in people's lives, at least in my experience.

Comment author: JamesAndrix 17 May 2009 07:46:27PM 0 points [-]

I identified 2 and 4 as most clearly about just wanting utilons.

I suspect that all such metawants can be reduced to trade-offs in the world. There is a bad tasting alertness potion, an addictive happiness drug with side effects, and a nonintuitive offer of money.

3 is a bit harder to look at this way, I think any solution needs to work just as well for Darryl who is also homosexual and finds it repulsive, but instead desires to no longer find it repulsive.

Comment deleted 19 May 2009 07:49:29AM *  [-]
Comment author: Alicorn 19 May 2009 02:33:45PM 1 point [-]

I'm sure you thought this would be cute or funny or something, but the objections aren't commensurate. I wasn't making a sweeping statement about the alertness-giving properties of Mountain Dew. Trying to do that would have been beside the point, since I was giving an individual example about my own beverage-related limitations, and living where I live, a Mountain Dew that might materialize in my home would have plenty of caffeine. To compare, I wouldn't have blinked if Psychohistorian had phrased the original remark about women as "I'll still find women alluring", making it about himself instead of about women.

Alternatively, I could just protest that in my idiolect, Mountain Dew refers to a beverage that contains caffeine, and your wacky foreign Mountain Dew is not Mountain Dew at all.

Comment deleted 19 May 2009 03:08:34PM *  [-]
Comment author: Cyan 20 May 2009 05:27:13AM *  3 points [-]

Where you "attribute [your] distraction entirely to the sense that it was directed at a presumed male audience", I attribute my distraction entirely to the sense that it was directed at a presumed American audience.

Here's where your analogy runs off the rails. Alicorn's text isn't directed at a presumed American audience -- it's directed at an audience presumed to be able to infer that Mountain Dew contains caffeine where she lives. Your rejoinder skips over this exact point made by Alicorn:

I wouldn't have blinked if Psychohistorian had phrased the original remark about women as "I'll still find women alluring", making it about himself instead of about women.

Comment deleted 20 May 2009 05:47:06AM [-]
Comment author: JGWeissman 20 May 2009 06:19:44AM 2 points [-]

The difference is that Phycohistorian was describing experiences that he intended the audience to recognize and identify with as their own, while Alicorn was describing her own experience as her own unique experience.

Comment deleted 20 May 2009 07:06:29AM [-]
Comment author: SoullessAutomaton 20 May 2009 09:37:15AM *  2 points [-]

It isn't necessarily a deliberate, conscious intent. However:

I know that Mountain Dew contains caffeine and that caffeine will make me alert. However, I also know that I hate Mountain Dew.

vs.

It's part of that set of things that doesn't go away no matter what you say or think about them. Women will still be alluring, food will still be delicious, and Michaelangelo's David will still be beautiful, no matter how well you describe these phenomenon.

Surely you see the difference?

Comment author: Alicorn 19 May 2009 03:24:51PM *  2 points [-]

I'm sorry if it wasn't clear, but the choice of the words "wacky foreign" was to be silly (in an apparently failed attempt to keep the discussion light), not to indicate an actual belief about the relative wackiness of foreign and domestic soft drinks.

Comment author: SoullessAutomaton 19 May 2009 09:46:56AM *  0 points [-]

Surely substituting 'Coke' or 'Pepsi' would make the Australians and Canadians among us feel more welcome.

This actually loses something in context, though--Mt. Dew (in the USA) has a somewhat higher caffeine content than those (about 20% more, I think), and also has a reputation as something people drink primarily for the caffeine content, not the flavor.

A better example might be "energy drinks" like Red Bull, which are typically dense, syrupy carbonated beverages with twice the caffeine content of a cola, but I'm not sure how common those are in other areas.

EDIT: This comment was written on the premise that the parent was a genuine request for greater acknowledgement of LW readers not in the USA, not a disingenuous attempt to make an off-topic point about something completely unrelated to this post. Disregard this comment as appropriate.

Comment deleted 20 May 2009 12:43:20PM [-]
Comment author: SoullessAutomaton 20 May 2009 07:34:51PM 1 point [-]

The grandparent was a genuine request. The sincerity of the disingenuous EDIT in the reply I have ironic doubts about.

I reevaluated the comment after seeing the downvotes it received and based on the apparent attempt to score a point in a discussion on a different post. If you were being genuine then okay, I accept your word on the matter and retract the edit with my apologies.

Comment author: Nick_Tarleton 20 May 2009 03:32:29AM *  1 point [-]

There's a big difference between parochialism that is, at worst, confusing, and parochialism that makes some people feel ignored or excluded (even if they aren't being ignored or excluded).

Comment deleted 20 May 2009 05:35:06AM *  [-]
Comment author: SoullessAutomaton 20 May 2009 09:49:50AM 2 points [-]

politically correct

You realize that this is an anti-applause light that conveys little informational value, right?

The actual argument is that since women comprise half the population and are severely underrepresented on LW as it is, if phrasing that implies the audience is uniformly male makes women feel excluded it is detrimental to the goal of spreading rationality. Do you actually have an argument against this?

Please note that "women won't actually feel excluded" is demonstrably false.

Comment deleted 20 May 2009 12:24:45PM *  [-]
Comment author: SoullessAutomaton 20 May 2009 07:40:16PM 2 points [-]

No. It adequately serves as a descriptive reference to the social dynamics involved in determining what is Right, moral, acceptable enlightened or otherwise good. If you can suggest a substitute phrase then I would happily adopt it. Arguments along the lines of 'something to do with political correct therefore something bad about the other side' are common. It is to be expected that some will assume a similar error of reasoning is being applied whenever the phrase is being used no matter the actual content and I would prefer to have a phrase that avoided this hassle.

The phrase you're looking for is probably "socially acceptable" or "social norm". The phrase "politically correct" is primarily used as a connotationally-loaded derogatory for social norms the speaker disagrees with, and to signal, in a beliefs-as-attire manner, group membership with certain political positions. If you want to criticize social norms, which I agree is a rewarding and enjoyable hobby, you would do better to name the specific norms you take issue with, rather than using a catch-all term for norms you dislike.

For instance: Which claims of exclusion are not acceptable to raise that you think ought to be? Why? What norms would you prefer?

I reject posts and certain of the normative demands contained therein on their own merit as I see it.

Okay. Which specific normative demands are you rejecting?

It is reasonable to desire an inclusive environment independently of the influence this improved environment may have on the spread of rationality. I for one accept inclusiveness a terminal value while 'spreading rationality' is not a goal of mine at all.

So you accept inclusiveness as valuable, but disagree with explanations given by individuals who felt excluded for why they felt that way, and feel that others are neglecting the spirit of mutual understanding across a cultural barrier? I'm not sure I follow.

Comment deleted 21 May 2009 12:54:42AM *  [-]
Comment author: SoullessAutomaton 21 May 2009 09:35:34AM *  1 point [-]

Do not presume so much.

Then I confess I am at a loss as to what your point in all this was, as you seem to have stated is a rejection of something that other people said without any real explanation as to what you're rejecting, or why.

Comment deleted 24 June 2009 01:36:42AM [-]
Comment author: SoullessAutomaton 24 June 2009 02:14:13AM *  6 points [-]

I don't normally remark on such things, but I'm a bit discouraged to note the following:

  • The parent comment was the first time Cameron_Taylor has posted anything in roughly a month, in a long-dead argument in which he and I were disagreeing.
  • At roughly the same time the parent comment was posted, roughly the last 80 or so posts I've made were all voted down, consecutively, once each, for no discernable reason.
  • I recall at least two other commenters mentioning being voted down suddenly on multiple unrelated comments previously while arguing with Cameron_Taylor.

Karma is easy-come, easy-go, but I'm thinking that someone is not exactly participating in good faith here.

Comment author: Drahflow 17 May 2009 08:30:40PM 0 points [-]

In a perfectly rational agent, no n-th order wants should exist.

Your problems with mountain dew might account for -1 util, you being awake for 2 utils, then you "want" to drink that stuff. Shut up and add.

The only source of multi-level desires I can see is an imperfect caching algorithm, which spews forth "Do not drink mountain dew" although the overall utility would be positive.

Comment author: latanius 18 May 2009 08:20:50AM 2 points [-]

Or the vertical hierarchy in our brains, and our inability to modify it. All four examples show "first order desires" created by lower level subsystems (taste perception, sexuality, motivations, etc.), and the higher level ones are trying to override it (consciousness, planning, models of the future world). But we aren't designed rationally enough to be able to control those lower levels, although we know partially about what they "think". ("We" = the conscious part writing LW comments...)

Comment author: conchis 18 May 2009 09:57:26AM *  1 point [-]

Your problems with mountain dew might account for -1 util, you being awake for 2 utils, then you "want" to drink that stuff. Shut up and add.

It still seems perfectly reasonable for a rational agent to not-want to not-want Mountain Dew here. If it were feasible to self-modify to become Moutain-Dew-indifferent at a cost less than 1 util, then the utility of drinking Mountain Dew & being indifferent to it would be greater than the utility of drinking it while continuing to not-want it.

Comment author: steven0461 18 May 2009 10:17:26AM *  1 point [-]

Self-modifying not to feel discomfort from drinking Mountain Dew isn't at all the same thing as self-modifying to want to drink Mountain Dew keeping discomfort constant. The latter isn't something you want to do on pain of committing the Wirehead Fallacy. The former isn't something that "want to want" language really applies to, as far as I can see.

Comment author: Alicorn 18 May 2009 02:45:58PM 1 point [-]

I don't see why not. Spicy foods inflict pain; wanting to develop a greater tolerance for cayenne pepper wouldn't be all that weird. Mountain Dew inflicts pain; learning to put up with that would be instrumentally useful to me. I'd prefer that it just not inflict pain, but assuming that's not an option, I'd be okay with just developing the ability to deal with it.

Comment author: stcredzero 18 May 2009 04:23:15PM *  2 points [-]

In case it hasn't been mentioned, many stores that sell Mountain Dew also stock NoDoz tablets, which you can develop a technique for swallowing whole with water.

I was actually in the position of wanting to develop a tolerance for spicy foods when I was young. I would go to the pantry and treat myself by dosing myself with Tabasco sauce. But I don't think of that as my "wanting to want KimChee." I wanted to eat KimChee like my parents and I took steps to achieve that goal.

I don't think there's any need for meta-desires at all, except for one: a drive to resolve conflicting desires. And this is arguably also a 1st order desire. There are obvious reasons why we'd evolve such a drive. We can explain away things like wanting to want internal conflict as merely a desire to be someone like Timothy Levitch from "Waking Life." It also makes sense that one can formulate paradoxical desires that are strongly resistant to resolution, or rationalize to oneself that they are conundrums. But I posit that these are merely ill-formed desires -- that if you can't reduce everything down to conflicting 1st-order desires, you haven't delved deeply enough.

http://dannyman.toldme.com/2003/11/19/timothy-levitch-waking-life/

Comment author: steven0461 18 May 2009 09:11:21PM *  0 points [-]

Agreed, what I said wasn't literally true. If you get 2 utils from caffeine and -1 util from pain, and if despite this you don't want to drink MD, then it's rational to self-modify to want to drink MD. But the point I meant to make is that it's not rational to self-modify to assign 0 utils to the same pain instead (because you don't care about utils, you care about things using utils), which is what I (mis)interpreted conchis as saying.

Comment author: conchis 18 May 2009 10:51:26AM 0 points [-]

I interpret "wanting to want" as encompassing the former, and don't really see any reason to limit it in the way you suggest. But either way, we have no substantive disagreement.

Comment author: MugaSofer 26 October 2012 08:49:25AM 0 points [-]

Carbonated beverages make my mouth hurt. I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.

Wait, that's unusual? I used to have the exact same problem, but I thought it was due to generalized willpower issues. When I got better at willpower, the problem disappeared (although I still tend to choose non-carbonated versions of drinks I like if possible.)

Comment author: Curiouskid 26 May 2011 12:06:13AM *  0 points [-]

I'm glad you introduced me to the term meta-wanting because it reminds me on an argument against free will.

Basically, you can go to a CD store (itunes now) and you can choose which CD you choose to buy because you prefer that CD. But you cannot prefer to prefer that CD. You simply prefer (1st order) that CD. You could try to raise the order of your preferences (an idea that had not occurred to me until now), but at the next highest order, your decision has already been made.

To me, that is the most convincing argument against free will that I've ever come across. Has anyone heard it before?

Comment author: MathieuRoy 23 November 2013 03:51:51PM *  1 point [-]

I want that my highest metawanting be this sentence.

  1. This is my highest order of metawanting.
  2. It was determine by me wanting it (so it wasn't already made).

I'm joking. I don't really want to want that my highest metawanting be wanting that my highest metawanting be wanting that my highest metawanting be wanting that.... haaaaaaaaaaaaaaaaaaaaa. :-)

Comment author: shminux 04 January 2012 03:08:30AM 0 points [-]

Have you considered that the free will debate is vacuous, as, ironically, we have no choice but to act as if we had free will?

Comment author: Curiouskid 04 January 2012 03:13:50AM 0 points [-]

So, I made this post before I'd even read the sequences.

Comment author: JamesCole 18 May 2009 10:33:50AM 0 points [-]

I don't think the right way to clarify this problem is by looking at it terms of first- and second-level desires. I think you need to turn it around and see it as a matter of what 'true self' means.

If people say that the desires you "endorse" on the second level are most reflective of your true self, they're wrong. This is because what we take to define our true selfs is based on different criteria, and those criteria define it such that people's second-level desires don't always match up with what we taken their 'true selves' to be, as in the case of Larry.

Comment author: dclayh 16 May 2009 03:46:56AM *  0 points [-]

So far so good. I look forward to the hard stuff :) And thanks for engaging my request.

Actually, your calling second-level agreement "endorsement" has led me to wonder whether there's a special term for desires that you want to want, want to want to want, and so on ad infinitum, analogous to common knowledge or Hofstadter's hyperrational groups (where everyone knows that everyone knows etc. that everyone is rational).

Comment author: Alicorn 16 May 2009 04:01:50AM 1 point [-]

I haven't run into any special jargon for endorsed desires, but it would be a cool word to have. There's some debate about whether we can really go up indefinitely in higher orders - it's not clear that we have the necessary cognitive capacity to go higher than about six nested intentional states. (For instance, I intend that you know that I intend that you know that I want to want Mountain Dew, but if you could come up with a more complicated string of propositional attitudes, I'd be a bit lost.)

Comment author: stcredzero 18 May 2009 04:52:02PM *  0 points [-]

I see nothing more than the intent to share knowledge of intent, or more concisely, to share intent. This can be reduced to a desire for synchrony of consciousness. When I sit down to play music with others, I do not have all of these intents and meta-intents. Yet it's arguably true that I do intend for the other musician to know that I intend him or her to know that I want to make good music. One can compose an infinite number of such statements concerning meta-intent and knowledge. But this is only an effect of the linearity of language, causing us to consider each constituent relationship one at a time. Beyond this, one can simply deal with the whole (jam session) in a highly functional, immediate, and mutually pleasing way.

Nth order desires are really just a variation on Xeno's paradox. It's a nifty mental exercise, but reality doesn't work that way. One just has 1st order wants, and a conflict resolution drive which is the only 2nd order want.

Comment author: Peter_de_Blanc 16 May 2009 05:41:12PM 1 point [-]

[I] wonder whether there's a special term for desires that you want to want, want to want to want, and so on ad infinitum

"Reflectively consistent."

Comment author: dclayh 16 May 2009 06:10:50PM *  0 points [-]

I think that's for beliefs, not desires.

Particularly because you can bring your beliefs and metabeliefs, etc., into alignment by reflection, whereas making your desires consistent requires at a minimum some kind of action, and may not be possible at all (except for the trivial case of instrumental second-order desires as in Ex. 1).

Comment author: dclayh 05 June 2009 03:29:53AM 0 points [-]

Update: upon reading Frankfurt I find that he calls this sort of infinitely regressed metawanting a "decisive" want or desire.

Comment author: cousin_it 16 May 2009 09:49:15AM 0 points [-]

Seems to be related to FAI. I'd look forward very much to any mathematic formalization of the term, if you have ideas how to get there.