I suspect most cases of "wanting to want" are better described as cases of internal conflict, where one part of us wishes that there weren't other parts of us with different conflicting wants.
Particularly where one part is responsible for the "internal narrative" and the other is responsible for motivation and prioritization, because the latter usually wins out and the former complains loudest.
Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.
Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.
I'm really bothered by my inability to see how to distinguish between these two classes of meta-wants. I suppose you just punt it off to your moral system, or your expected-value computations.
Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn't really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn't the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn't wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.
Mimi the Heroin Addict, by contrast, probably doesn't want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin's negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.
In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.
The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results "Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal." I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of...
Let me try using an extended metaphor to explain my point: Remember Eliezer's essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?
Let's imagine a race of Pebblesorters that's p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.
Now let's suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn't a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.
The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn't a prime number. They begin to persecute 11-Pebble Fav...
We should point people to this whenever they're like "What's special about Less Wrong?" and we can be like "Okay, first, guess how Less Wrong would discuss a reluctant Christian homosexual. Made the prediction? Good, now click this link."
I upvoted despite this. If you overlook that one problem, everything else is gold. That single flawed sentence does not effect the awesome of the other 14 paragraphs, as it does not contribute to the conclusion.
if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I'm sure you could think of some things to write in the "pro" column for "take the pill" and in the "con" column for "don't take the pill."
I don't deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.
However, I would maintain that the specific reason given in Alicorn's orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.
...It's also worth considering how much one wants to engage in sour grapes th
Harry Frankfurt, who came up with the original idea, did a much better job in explaining in my opinion. (Why are you not referring to his paper?)
Here is the link for the curious: http://www.usfca.edu/philosophy/pdf%20files/Freedom%20of%20the%20Will%20and%20the%20Concept%20of%20a%20Person.pdf
It's not always so easy to say which desire is actually first order and which is second order.
For instance, example 3 could be inverted:
Larry was brought up to believe God hates homosexuality. Because of this he experiences genuine disgust when he thinks about homosexual acts, and so desires not to perform them or even think about them (first order). However, he really likes his friend Ted and sometimes wished God wasn't such a dick (second order).
There's likely even a third order desire: Larry was brought up to be a good Christian, and desperately wishes...
There's a simpler model for all of these examples -- you're describing conflicts between an "away-from" motivation and a "towards" motivation. These systems are semi-independent, via affective asynchrony. The second-order want is then arising as a subgoal of the currently-active goal (be alert, etc.).
I guess what I'm trying to say here is that there really aren't "second order wants" in the system itself; they're just an emergent property of a system with subgoals that explicitly models itself as an agent, especially if it a...
Carbonated beverages make my mouth hurt. I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.
Wait, that's unusual? I used to have the exact same problem, but I thought it was due to generalized willpower issues. When I got better at willpower, the problem disappeared (although I still tend to choose non-carbonated versions of drinks I like if possible.)
I'm glad you introduced me to the term meta-wanting because it reminds me on an argument against free will.
Basically, you can go to a CD store (itunes now) and you can choose which CD you choose to buy because you prefer that CD. But you cannot prefer to prefer that CD. You simply prefer (1st order) that CD. You could try to raise the order of your preferences (an idea that had not occurred to me until now), but at the next highest order, your decision has already been made.
To me, that is the most convincing argument against free will that I've ever come across. Has anyone heard it before?
I don't think the right way to clarify this problem is by looking at it terms of first- and second-level desires. I think you need to turn it around and see it as a matter of what 'true self' means.
If people say that the desires you "endorse" on the second level are most reflective of your true self, they're wrong. This is because what we take to define our true selfs is based on different criteria, and those criteria define it such that people's second-level desires don't always match up with what we taken their 'true selves' to be, as in the case of Larry.
In a perfectly rational agent, no n-th order wants should exist.
Your problems with mountain dew might account for -1 util, you being awake for 2 utils, then you "want" to drink that stuff. Shut up and add.
The only source of multi-level desires I can see is an imperfect caching algorithm, which spews forth "Do not drink mountain dew" although the overall utility would be positive.
So far so good. I look forward to the hard stuff :) And thanks for engaging my request.
Actually, your calling second-level agreement "endorsement" has led me to wonder whether there's a special term for desires that you want to want, want to want to want, and so on ad infinitum, analogous to common knowledge) or Hofstadter's hyperrational groups (where everyone knows that everyone knows etc. that everyone is rational).
Suppose also that there is a can of Mountain Dew handy. I know that Mountain Dew contains caffeine and that caffeine will make me alert.
I am hesitant to bring it up because I don't want to become the multiculturalism police on LessWrong, but I found this distracting. American Mountain Dew contains a large caffeine content yet in most other countries Mountain Dew is Caffeine free. There is a significant minority of LessWrong participants who do not dwell in America and those readers can not help but become distracted when posts seem to be clearly inten...
It's not that I think there's literally a math equation locked in the human psyche that encodes morality. It's more like there are multiple (sometimes conflicting) moral values and methods for resolving conflicts between them and that the sum of these can be modeled as a large and complicated equation.
I'm sorry, there's an ambiguity there - when you say "the sum of these", are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
You gave me the impression that Marvin valued "staying alive" less as an end in itself, and more as a means to achieve the end of improving the world. in particular when you said this:
You are quite correct. I apologise; I changed that example several times from where I started, and it seems that one of my last-minute changes actually made it a worse example (my aim was to try to show how the explicit aim of self-preservation could be a reasonable moral aim, but in the process I made it not a moral aim at all). I should watch out for that in the future.
This is actually something that bothers me in fiction when a character who is superhumanly good and power (i.e. Superman, the Doctor) risks their lives to save a relatively small amount of people. It seems short-sighted of them to do that since they regularly save much larger groups of people and anticipate continuing to do so in the future, so it seems like they should preserve their lives for those people's sakes.
I've always felt that was because one of the effects of great power, is that it's so very easy to let everyone die. With great power, as Spiderman is told, comes great responsibility; one way to ensure that you're not letting your own power go to your head, is by refusing to not-rescue anyone. After all, if the average science hero lets everyone he thinks is an idiot die, then who would be left?
Sometimes there's a different reason, though; Sherlock Holmes would ignore a straightforward and safe case to catch a serial killer in order to concentrate on a tricky and deadly case involving a stolen diamond; he wasn't in the detective business to help people, he was in the detective business in order to be challenged, and he would regularly refuse to take cases that did not challenge him.
(That's probably a fair example as well, actually; for Holmes, only the challenge, the mental stimulation of a worthy foe, is important; for Superman, what is important is the saving of lives, whether from a mindless tsunami or Lex Luthor's latest plot).
I think that we, and most humans, have the same basic desires, where we differ is the object of those desires, and the priority of those desires.
Hmmm. If you're willing to accept zero, or near-zero, as a priority, then that statement can apply to any two sets of desires. Consider Sherlock holmes and a paperclipper; Holmes' desire for mental stimulation is high-priority, his desire for paperclips is zero-priority, while the paperclipper's desire for paperclips is high priority, and its desire for mental stimulation is zero-priority. (Some desires may have negative priority, which can then be interpreted as a priority to avoid that outcome - for example, my desire to immerse my hand in acid is negative, but a masochist may have a positive priority for that desire)
This implies that, in order to meaningfully differentiate the above statement from "some people have different desires", I may have to designate some very low priority, below which the desire is considered absent (I may, of course, place that line at exactly zero priority). Some desires, however, may have no priority on their own, but inherit priority from another desire that they feed into; for example, a paperclipper has zero desire for self-preservation on its own, but it will desire self-preservation so that it can better create more paperclips.
Now, given a pool of potential goals, most people will pick out several desires from that pool, and there will be a large overlap between any two people (for example, most humans desire to eat - most but not all, there are certain eating disorders that can mess with that), and it is possible to pick out a set of desires that most people will have high priorities for.
It's even probably possible to pick out a (smaller) set of desires such that those who do not have those desires at some positive priority are considered psychologically unhealthy. But such people nonetheless do exist.
Does this mean it isn't bad to oppose paperclipping? I don't know, maybe, but maybe not.
In my personal view, it is neutral to paperclip or to oppose paperclipping. It becomes bad to paperclip only when the paperclipping takes resources away from something more important.
And there are circumstances (somewhat forced circumstances) where it could be good to paperclip.
For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority.
This seems like a difference in priority, rather than desire, as most people would prefer differing proportions of both both. It's still a legitimate disagreement, but I think it's more about finding a compromise between conflicting priorities, rather than totally different values.
There exist people who would place negative value on the idea of following the instuctions of any legitimate authority. (They tend to remain a small and marginal group, because they cannot in turn form an authority for followers to follow without rampant hypocrisy).
Compounding this problem is the fact that people value diversity to some extent. We don't value all types of diversity obviously, I think we'd all like to live in a world where people held unanimous views on the unacceptability of torturing innocent people. But we would like other people to be different from us in some ways. Most people, I think, would rather live in a world full of different people with different personalities than a world consisting entirely of exact duplicates (in both personality and memory) of one person. So it might be impossible to reach full agreement on those other values without screwing up the achievement of the Value of Diversity.
Yes, diversity has many benefits. The second-biggest benefit of diversity is that some people will be more correct than others, and this can be seen in the results they get; then everyone can re-diversify around the most correct group (a slow process, taking generations, as the most successful group slowly outcompetes the rest and thus passes their memes to a greater and/or more powerful proportion of the next generation). By a similar tokem, it means that one something happens that detroys one type of person, it doesn't destroy everyone (bananas have a definite problem there, being a bit of a monoculture).
The biggest benefit is that it leads to social interaction. A completely non-diverse society would have to be a hive mind (or different experiences would slowly begin to introduce diversity), and it would be a very lonely hive mind, with no-one to talk to.
I'm sorry, there's an ambiguity there - when you say "the sum of these", are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
Nearly all of humanity as a whole. There are obviously some humans who don't really value morality, we call them sociopaths, but I think most humans care about very similar moral concepts. The fact that people have somewhat different personal preferences and desires at first might seem to challenge this idea, but I don't really think it does. It just means that there a...
In response to a request, I am going to do some basic unpacking of second-order desire, or "metawanting". Basically, a second-order desire or metawant is a desire about a first-order desire.
Example 1: Suppose I am very sleepy, but I want to be alert. My desire to be alert is first-order. Suppose also that there is a can of Mountain Dew handy. I know that Mountain Dew contains caffeine and that caffeine will make me alert. However, I also know that I hate Mountain Dew1. I do not want the Mountain Dew, because I know it is gross. But it would be very convenient for me if I liked Mountain Dew: then I could drink it, and I could get the useful effects of the caffeine, and satisfy my desire for alertness. So I have the following instrumental belief: wanting to drink that can of Mountain Dew would let me be alert. Generally, barring other considerations, I want things that would get me other things I want - I want a job because I want money, I want money because I can use it to buy chocolate, I want chocolate because I can use it to produce pleasant taste sensations, and I just plain want pleasant taste sensations. So, because alertness is something I want, and wanting Mountain Dew would let me get it, I want to want the Mountain Dew.
This example demonstrates a case of a second-order desire about a first-order desire that would be instrumentally useful. But it's also possible to have second-order desires about first-order desires that one simply does or doesn't care to have.
Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.
One thing that is often said is that what first-order desires you "endorse" on the second level are the ones that are your most true self. This seems like an appealing notion in Mimi's case; I would not want to say that at her heart she just wants heroin and that's an intrinsic, important part of her. But it's not always the case that the second-order desire is the one we most want to identify with the person who has it:
Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.
In this case, I think we're tempted to say that poor Larry is a gay guy who's had an alien second-order desire attached to him via his upbringing, not a natural homophobe whose first-order desires are insidiously eroding his real personality.
A less depressing example to round out the set:
Example 4: Suppose Olivia the Overcoming Bias Reader, whose very prescient mother predicted she would visit this site, is convinced on by Eliezer's arguments about one-boxing in Newcomb's Problem. However, she's pretty sure that if Omega really turned up, boxes in hand, she would want to take both of them. She thinks this reflects an irrationality of hers. She wants to want to one-box.
1Carbonated beverages make my mouth hurt. I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.