LESSWRONG
LW

Comment Permalink

CCC12y00

I don't think I'm coming across right. I'm not saying that morality is some sort of collective agreement of people in regards to their various preferences. I'm saying that morality is a series of concepts such as fairness, happiness, freedom etc., that these concepts are objective in the sense that it can be objectively determined how much fairness, freedom, happiness etc. there is in the world, and that the sum of these concepts can be expressed as a large equation.

This is probably true.

This equation isn't written into the human psyche; it exists independantly of what people think about morality. It just is. And even if we don't know exactly what the equation is, even if we can't work out the morality of a given act down to the tenth decimal place, we can still apply basic heuristics and arrive at a usable estimate in most situations.

My question is, then - assuming the above is true, how do we find that equation? Does there exist some objective method whereby you, I, a Pebblesorter, and a Paperclipper can all independently arrive at the same definition for what is moral (given that the Pebblesorter and Paperclipper will almost certainly promptly ignore the result)?

(I had thought that you were proposing that we find that equation by summing across the moral values and imperatives of humanity as a whole - excluding the psychopaths. This is why I asked about the exclusion, because it sounded a lot like writing down what you wanted at the end of the page and then going back and discarding the steps that wouldn't lead there; that is also why I asked about the aliens).

I don't know if I could tell, but I'd very much prefer that the AI not do that, and would consider myself to have been massively harmed if it did, even if I never found out. My preference is to actually interact with a diverse variety of people, not to merely have a series of experiences that seem like I'm doing it.

Yes, I think we're in agreement on that. (Though this does suggest that 'sentient' may need a proper definition at some point).

nshepperd12y10

What you're saying - and correct me if I'm wrong - is that there is some objective True Morality, some complex equation that, if applied to any possible situation, will tell you how moral a given act is.

In the same way as there exists a True Set of Prime Numbers, and True Measure of How Many Paperclips There Are...

0Ghatanathoah12y

Even though the equation exists independently of our thoughts (the same way primality exists independently from Pebblesorter thoughts) fact that we are capable of caring about the results given by the equation means we must have some parts of it "written" in our heads, the same way Pebblesorters must have some concept of primality "written" in their heads. Otherwise, how would we be capable of caring about its results? I think that probably evolution metaphorically "wrote" a desire to care about the equation in our heads because if humans care about what is good and right it makes it easier for them to cooperate and trust each other, which has obvious fitness advantages. Of course, the fact that evolution did a good thing by causing us to care about morality doesn't mean that evolution is always good, or that evolutionary fitness is a moral justification for anything. Evolution is an amoral force causes many horrible things to happen. It just happened that in this particular instance, evolution's amoral metaphorical "desires" happened to coincide with what was morally good. That coincidence is far from the norm, in fact, evolution probably deleted morality from the brains of sociopaths because double-crossing morally good people also sometimes confers a fitness advantage. So how do we learn more about this moral equation that we care about? One common form of attempting to get approximations of it in philosophy is called reflective equilibrium, where you take your moral imperatives and heuristics and attempt to find the commonalities and consistencies they have with each other. It's far from perfect, but I think that this method has produced useful results in the past. Eliezer has proposed what is essentially a souped up version of reflective equilibrium called Coherent Extrapolated Volition. He has argued, however, that the primary use of CEV is in designing AIs that won't want to kill us, and that attempting to extrapolate other people's volition is open to cor

See in context

30 Wanting to Want

by Alicorn

16th May 2009

2 min read

199

30

In response to a request, I am going to do some basic unpacking of second-order desire, or "metawanting". Basically, a second-order desire or metawant is a desire about a first-order desire.

Example 1: Suppose I am very sleepy, but I want to be alert. My desire to be alert is first-order. Suppose also that there is a can of Mountain Dew handy. I know that Mountain Dew contains caffeine and that caffeine will make me alert. However, I also know that I hate Mountain Dew¹. I do not want the Mountain Dew, because I know it is gross. But it would be very convenient for me if I liked Mountain Dew: then I could drink it, and I could get the useful effects of the caffeine, and satisfy my desire for alertness. So I have the following instrumental belief: wanting to drink that can of Mountain Dew would let me be alert. Generally, barring other considerations, I want things that would get me other things I want - I want a job because I want money, I want money because I can use it to buy chocolate, I want chocolate because I can use it to produce pleasant taste sensations, and I just plain want pleasant taste sensations. So, because alertness is something I want, and wanting Mountain Dew would let me get it, I want to want the Mountain Dew.

This example demonstrates a case of a second-order desire about a first-order desire that would be instrumentally useful. But it's also possible to have second-order desires about first-order desires that one simply does or doesn't care to have.

Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.

One thing that is often said is that what first-order desires you "endorse" on the second level are the ones that are your most true self. This seems like an appealing notion in Mimi's case; I would not want to say that at her heart she just wants heroin and that's an intrinsic, important part of her. But it's not always the case that the second-order desire is the one we most want to identify with the person who has it:

Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.

In this case, I think we're tempted to say that poor Larry is a gay guy who's had an alien second-order desire attached to him via his upbringing, not a natural homophobe whose first-order desires are insidiously eroding his real personality.

A less depressing example to round out the set:

Example 4: Suppose Olivia the Overcoming Bias Reader, whose very prescient mother predicted she would visit this site, is convinced on by Eliezer's arguments about one-boxing in Newcomb's Problem. However, she's pretty sure that if Omega really turned up, boxes in hand, she would want to take both of them. She thinks this reflects an irrationality of hers. She wants to want to one-box.

¹Carbonated beverages make my mouth hurt. I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.

Utility Functions

Personal Blog

30

Mentioned in

172Three ways CFAR has changed my view of rationality

41Gandalf or Saruman? A Soldier in Scout's Clothing

27Highlights and Shadows

New Comment

199 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:07 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]RobinHanson16y260

I suspect most cases of "wanting to want" are better described as cases of internal conflict, where one part of us wishes that there weren't other parts of us with different conflicting wants.

[-]SoullessAutomaton16y100

Particularly where one part is responsible for the "internal narrative" and the other is responsible for motivation and prioritization, because the latter usually wins out and the former complains loudest.

9nazgulnarsil16y

furthermore, the internal narrative has been carefully honed to be able to be disingenuous for signaling purposes.

3MichaelHoward16y

More to the point, the internal narrative part largely doesn't need to be disingenuous for signaling purposes, because it's kept in the dark about what the motivation and prioritization part is really up to.

6MichaelVassar16y

Agreed, but the parts in conflict may be of vastly different reflectivity. Some relevant parts may not have anything analogous to awareness of some other parts.

6Jack16y

Just so everyone is clear: That is one way of describing cases where second order desires conflict with first order desires, perhaps. But one can want to to want X and want X... its just that Alicorn used only examples where the two conflict (and probably the distinction is best illustrated by looking at the conflicts). But right now I have both a first order desire not to use heroin and a second order desire to not use heroin. In fact, the vast majority of our desires are probably like this. So most cases of "wanting to want" are not cases of internal conflict but perhaps these cases can be described as instances of internal consistency.

1stcredzero16y

In any case, I think Occam's Razor demands that we reject the notion of generalized second-order desires. We can leave that concept out entirely and explain everything as conflicts between first order desires and a generalized desire for consistency and/or resolution. Note that in all the examples, there are conflicting goals. In (1) it's the desire to stay awake vs. avoiding noxious stimuli. In (2) it's the desire to stay alive vs. cop another high. In (3) it's the desire to live up to his upbringing vs. follow his sexual urges. I'm not even sure that a generalized desire for consistency and/or resolution would be a second-order desire. I think that the feeling of conflict over not being able to decide which speaker to buy is a lot like resolving conflicts between incompatible desires. The only difference is that choosing to buy a speaker is usually morally neutral, but there is societal pressure to choose one option as the "right" one in 2 and 3, and an imperative to preserve one's life in 1 and 2, so we are steered towards a particular outcome. It may well be only a trick of language that prompts us to say "I want to want X." I suspect that we could also say, "I want X and I want Y. I cannot have both, and I know I'm supposed to want X. I wish I wasn't conflicted." But that is much longer than, "I want to want X."

2freyley16y

"Better" in what way? Do you mean better in that you think it's a more accurate view of the inside of your head? Or better in that it's a more helpful metaphorical view of the situation that can be used to overcome the difficulties described? I think the view of it as a conflict between different algorithms is useful, and it's the one that I start with, but I wonder whether different views of this problem might be helpful in developing more methods for overcoming it.

0JamesAndrix16y

The thing I'm getting from all this is: Any time you have two desires that turn out in the environment to be contradictory, you could also have a desire to change one of them (the 'lesser' one?) But we don't always get this desire to want something different, I'm wondering if always should, never should, or if there is some clear rule.

0steven046116y

Seconded; more specifically it seems to me that if one does not want something but one wants to want it, it has to be the case that either: * there's one entity doing the not wanting, and some other entity that wants the one entity to stop doing the not wanting * one values wanting the thing for the sake of wanting the thing or for the sake of some result of wanting the thing other than getting the thing, not for the sake of getting the thing, and moreover one values wanting the thing so much that this outweighs the extra likelihood of getting the thing * one does not want to want the thing unconditionally, but one does want the thing to turn into a different thing that one does want (e.g. Alicorn's Mountain Dew example; there the thing one wants is to enjoy and want to drink Mountain Dew, which is not the same thing as wanting to want to drink Mountain Dew even though one does not enjoy it) * "wanting" here is some more informal human thing that isn't captured by rational decision theory (which is bad!) Are those all the possibilities?

[-]PhilGoetz16y100

Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.

Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.

I'm really bothered by my inability to see how to distinguish between these two classes of meta-wants. I suppose you just punt it off to your moral system, or your expected-value computations.

[-]Ghatanathoah13y120

Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn't really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn't the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn't wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.

Mimi the Heroin Addict, by contrast, probably doesn't want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin's negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.

3Vaniver13y

What reasoning process did you use to determine his belief was mistaken? When and where does Larry live? What are his other terminal goals?

[-]Ghatanathoah13y100

In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.

The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results "Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal." I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of... (read more)

3Vaniver13y

PhilGoetz wrote: I am having trouble seeing a significant difference between that and what you've described. Mimi's enabler could argue "human happiness is a Good Thing unless it seriously interferes with some other really important goal," and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the "serious interference" line. Similarly, the illegality of heroin and the illegality of homosexuality seem similarly comparable; perhaps Mimi should convince her fellows that their intolerance of her behavior is unethical.

[-]Ghatanathoah13y350

Let me try using an extended metaphor to explain my point: Remember Eliezer's essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?

Let's imagine a race of Pebblesorters that's p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.

Now let's suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn't a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.

The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn't a prime number. They begin to persecute 11-Pebble Fav... (read more)

[-]Eliezer Yudkowsky13y130

We should point people to this whenever they're like "What's special about Less Wrong?" and we can be like "Okay, first, guess how Less Wrong would discuss a reluctant Christian homosexual. Made the prediction? Good, now click this link."

-3Epiphany13y

I'm surprised you regarded it so highly. The flaws I noticed are located in a response to Ghatanathoah's comment.

9Epiphany13y

First, I would like to make one thing clear: I have absolutely nothing against homosexuals and in fact qualify as queer because my attractions transcend gender entirely. I call my orientation "sapiosexual" because it is minds that I am sexually attracted to, and good character, never mind the housing. Stops at "pigheaded jerks" downvotes You know where this is going, oh yes, I am going right to fundamental attribution error and political mindkill. The parents are deemed "pigheaded jerks" - a perception of their personality. Larry the homosexual, convinced by the exact same reasoning, is given something subtly different - an attack on his behavior -- "he gullibly believed them" and you continue with "They (the Larrys) just think they are because their parents fed them a load of crap." attributing his belief to the situation that Larry is in. Do you think Larry's grandparents didn't teach Larry's parents the same thing? And that Larry's great grandparents didn't teach it to Larry's grandparents? This was a "good solid dig" at the other side.

[-]MugaSofer13y100

I upvoted despite this. If you overlook that one problem, everything else is gold. That single flawed sentence does not effect the awesome of the other 14 paragraphs, as it does not contribute to the conclusion.

-1Epiphany13y

My experience of it was more like: "Oh, this is nice and organized... Still orderly... Still orderly... OHMYSPAGHETTIMONSTER I DID NOT JUST READ THAT!" To me, it was a disappointment. Like if I were eating ice cream and then it fell to the ground. If Eliezer is going to praise it like it's the epitome of what LessWrong should be, then it should be spotless. Do you agree?

7Ghatanathoah13y

You make an excellent point. I will edit my post to make it sound less political and judgemental.

4Epiphany13y

I am charmed by your polite acknowledgement of the flaw and am happy to see that this has been updated. Thanks for letting me know that pointing it out was useful. :)

6Vaniver13y

I think you're looking at this discussion from the wrong angle. The question is, "how do we differentiate first-order wants that trump second-order wants from second-order wants that trump first-order wants?" Here, the order only refers to the psychological location of the desire: to use Freudian terms, the first order desires originate in the id and the second order desires originate in the superego. In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.) Some cases are easier than others- in the case of Sally, who wants to commit suicide but wants to not want to commit suicide, I would generally recommend methods of effective treatment for suicidal tendencies, not the alternative. But you should be able to recognize that the decision could be difficult, at least for some alteration of the parameters, and is the alteration is significant enough it could swing the other way. There is also another factor which clouds the analysis, which is that the ego has to weigh the costs of altering, suppressing, or foregoing one of the desires. It could be that Larry has a twin brother, Harry, who is not homosexual, and that Harry is genuinely happier that Larry is, and that Larry would genuinely prefer being Harry to being himself; he's not mistaken about his second-order want. However, the plan to be (or pretend to be) straight is much more costly and less likely to succeed than the plan to stop wanting to be straight, and that difference in costs might be high enough to determine the ego's decision. Again, it should be possible to imagine realistic cases in which the decision would swing the other way. (Related.) It's also worth considering how much one wants to engage in sour grapes thinking-

[-]Ghatanathoah13y100

if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I'm sure you could think of some things to write in the "pro" column for "take the pill" and in the "con" column for "don't take the pill."

I don't deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.

However, I would maintain that the specific reason given in Alicorn's orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.

It's also worth considering how much one wants to engage in sour grapes th

... (read more)

0JaySwartz12y

While the Freudian description is accurate relative to sources, I struggle to order them. I believe it is an accumulated weighting that makes one thought dominate another. We are indeed born with a great deal of innate behavioral weighting. As we learn, we strengthen some paths and create new paths for new concepts. The original behaviors (fight or flight, etc.) remain. Based on this known process, I conjecture that experiences have an effect on the weighting of concepts. This weighting sub-utility is a determining factor in how much impact a concept has on our actions. When we discover fire burns our skin, we don't need to repeat the experience very often to weigh fire heavily as something we don't want touching our skin. If we constantly hear, "blonde people are dumb," each repetition increases the weight of this concept. Upon encountering an intelligent blond named Sandy, the weighting of the concept is decreased and we create a new pattern for "Sandy is intelligent" that attaches to "Sandy is a person" and "Sandy is blonde." If we encounter Sandy frequently, or observe many intelligent blonde people, the weighting of the "blonde people are dumb" concept is continually reduced. Coincidentally, I believe this is the motivation behind why religious leaders urge their followers to attend services regularly, even if subconsciously. The service maintains or increases weighting on the set of religious concepts, as well as related concepts such as peer pressure, offsetting any weighting loss between services. The depth of conviction to a religion can potentially be correlated with frequency of religious events. But I digress. Eventually, the impact of the concept "blonde people are dumb" on decisions becomes insignificant. During this time, each encounter strengthens the Sandy pattern or creates new patterns for blondes. At some level of weighting for the "intelligent" and "blonde" concepts associated to people, our brain economizes by creating a "blond people are i

6NancyLebovitz12y

From what I've heard, the typical response to believing that blond people are dumb and observing that blond Sandy is intelligent is to believe that Sandy is an exception, but blond people are dumb. Most people are very attached to their generalizations.

0JaySwartz12y

Quite right about attachment. It may take quite a few exceptions before it is no longer an exception. Particularly if the original concept is regularly reinforced by peers or other sources. I would expect exceptions to get a bit more weight because they are novel, but no so much as to offset higher levels of reinforcement.

2CCC13y

Your example does an exemplery job of explaining your viewpoint on Larry's situation. To explain the presumed viewpoint of Larry's parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9. How do you define objective morality? I've heard of several possible definitions, most of which conflict with each other, so I'm a little curious as to which one you've selected.

4Ghatanathoah13y

I'm not sure I understand you, do you mean that a more precise description of Larry's parent's viewpoint is that the Pebblesorter versions of them think 11 and 9 are the same numbers? Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument. I define morality as being a catch-all term to describe what are commonly referred to as the "good things in life," love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things. In other words, "good" and "right" are synonyms. Morality is objective because we can objectively determine whether people are happy, being treated fairly, getting what they want out of life, etc. In Larry's case having a relationship with Ted-the-Next-Door neighbor would be the morally right thing to do because it would increase the amount of love, happiness, people-getting what they want, etc. in the world. I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism. That is, they believe that moral knowledge is intrinsically motivating, simply knowing something is right motivates someone to do it. They then conclude that since intrinsically motivating knowledge doesn't seem to exist, that morality must be subjective. I am a motivational externalist, so I do not buy this argument. I believe that that people are motivated to act morally by our conscience and moral emotions (i.e. compassion, sympathy). If someone has no motivation to act to increase the "good things in life" that doesn't mean morality is subjective, that simply means that they are a bad person. People who lack moral emotions exist in real life, and they seem to lack any desire to act morally at all, unless you threaten to punish them if they don't. The idea of intrinsically motivating knowledge is pretty scary if you think about

5CCC13y

Yes, exactly. Larry's parents' do not believe that they are mistaken, and are not easily proved mistaken. That's a good definition, and it avoids most of the obvious traps. A bit vague, though. Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative... except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I'd still think that leaving the child locked away in the dark is a wrong thing. (This can also lead to Pascal's Mugging, as an edge case) In my case, it's because every attempt I've seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche? There are some bad things in the dark corners of the human psyche. Take whatever action is most likely to transform the world into a paradise? Doesn't usually work, because we don't know enough to always select the correct actions. Do unto others as you would have them do unto you? That's a very nice one - but not if Bob the Masochist tries to apply it. Of course, subjective morality is no better - and is often worse (mainly because a society in general can reap certain benefits from a shared idea of morality). What does seem to work is to pick a society whose inhabitants seem happy and fulfilled, and trying to use whatever rules they use. The trouble with that is that it's kludgy, uncertain, and could often do with improvement (though it's been improved often enough in human history that many - not all, but many - obvious 'improvements' aren't in practice).

9Nornagest13y

Aside from its obvious artificiality, and despite the fact that all our instincts cry out against it, it's not at all clear to me that there are any really good reasons to reject the Omelasian solution. This is of course a fantastically controversial position (just look at the response to Torture vs. Dust Specks, which might be viewed as an updated and reframed version of the central notion of The Ones Who Walk Away From Omelas), but it nonetheless seems to be a more or less straightforward consequence of most versions of consequential ethics. As a matter of fact, I'm inclined to view Omelas as something between an intuition pump and a full-blown cognitive exploit: a scenario designed to leverage our ethical heuristics (which are well-adapted to small-scale social groups, but rather less well adapted to exotic large-scale social engineering) in order to discredit a viewpoint which should rightfully stand or fall on pragmatic grounds. A tortured child is something that hardly anyone can be expected to think straight through, and trotting one out in full knowledge of this fact in order to make a point upsets me.

8satt12y

There's a real world analogue to Omelas. The UK (like other countries) has a child protection system, intended to minimize abuse & neglect of children. The state workers (health visitors, social workers, police officers, hospital staff, etc.) who play roles in the system can try to intervene when the apparent risk of harm to a child reaches some threshold. If the threshold is too low, the system gratuitously interferes with families' autonomy, preventing them from living their lives in peace. If the threshold is too high, the system fails to do its job of preventing guardians from tormenting or killing their children. Realistically, a trade-off is inevitable, and under any politically feasible threshold "some children will die to preserve the freedom of others", as the sociologist Robert Dingwall put it. So the UK's child protection system takes the Omelasian route. The real life situation is less black & white than Omleas, but it looks like the same basic trade-off in a non-artificial setting. I wonder whether people's intuitions about Omelas align with their intuitions about the real life child protection trade-off (and indeed whether both align with society's revealed collective preference).

5Nornagest12y

I'd agree that these are consequentially similar, but I don't think they're psychologically similar at all. There's an element of exploitation in Omelas that isn't present in social services: state workers are positioned as protecting children from an evil unrelated to the state, while Omelas is cast as willfully perpetrating an evil in order to ensure its own prosperity. People tend to think of moral culpability in terms of blame, and although some blame might attach itself to social workers for failing to stop abuses that they might prevent with more intrusive intervention thresholds, it's much diluted by the vastly more viscerally appalling culpability carried by actual abusers. Omelas offers no subjects for condemnation other than the state apparatus and the citizens supporting it. On top of that, intrusive child protection services have very salient failings: most parents (and most children) would find government intrusion into their family lives extremely unpleasant, unpleasant enough to fear and take political action to avoid. Meanwhile, the consequences of no longer torturing Omelas' sacrificial lamb are unspecified and thus about as far from salient (re-entrant?) as it's possible to get. Even in a hypothetical fully specified Omelas where we could point to a chain of effects, I'd expect that chain to be a lot longer and harder to follow, and its endpoints hence less emotionally weighty.

0Multiheaded12y

Link to me mentioning both Omelas and another "eternally tortured child" short story, SCP-231 (potentially highly distressing so I'm not hotlinking it), as an intuition pump against Mencius Moldbug's "Patchwork" proposal (the "strong"/"total" vision of Patchwork, with absolute security of sovereigns) along very similar lines of analogy, over in the Unqualified Reservations comments.

3[anonymous]12y

Disagree SCP-231 is bad source of intuition because it is crafted to be torture & horror porn.

3Multiheaded12y

There are many, many things in the similarly exploitative franchise known as "Real Life" that also appear to be crafted as "torture & horror porn". So I don't see the problem with linking to a fictionalized version. I dare say that any story without elements that induce horror and revulsion in a reader would be an inadequate source of intuition for considering the most shocking aspects of our own world... or the ethics of knowingly creating a system which offers absolute security indiscriminately to those who would create such nightmares and those who'd seek to prevent them. Example of a victim testimony 1: - trigger warnings for extreme child abuse, rape, pedophilia and psychological damage. Example of a victim testimony 2: - all of the above, except even more outspoken descriptions of the author's mental anguish. (NSFanywhere. The main blog has... images... that are more gore than extreme porn; don't look unless you're massively desensitized.) Want some amnesiacs yet? You might be able to forget those stories faster if you don't think about the fact that something similar must be happening somewhere in your country, probably in your city, at this very moment. Oops, too late! (Again, sorry for the confrontational tone and such - I wanted to hammer home the point that sometimes it's the violently emotional reaction to an objectively terrible problem that would be true to your desires, and trying to stay "detached" and "reasonable" would be self-deception. See: deathism.)

4[anonymous]12y

Indeed. But in the context of the discussion the story primes you to live one kind of horror and not another when making trade offs between the two. This is why I objected to it.

2JaySwartz12y

I struggle with conceiving wanting to want, or decision making in general, as a tiered model. There are a great many factors that modify the ordering and intensity of utility functions. When human neurons fire they trigger multiple concurrent paths leading to a set of utility functions. Not all of the utilities are logic-related. I posit that our ability to process and reason is due to this pattern ability and any model that will approximate human intelligence will need to be more complex than a simple linear layer model. The balance of numerous interactive utilities combine to inform decision making. A multiobjective optimization model, such as PIBEA, is required. I'm new to LW, so I can't open threads just yet. I'm hoping to find some discussions around evolutionary models and solution sets relative to rational decision processing.

1Multiheaded12y

I sort of agree with your comment, but you should've probably posted it as a reply to the main post, not to this specific comment of mine, as the topics seem to be somewhat apart. (BTW you can open threads in Discussion just fine seeing as you're at 10 karma - go ahead!)

2DaFranker12y

Disagree. Do look, even if it hurts. Especially if it hurts a lot (if it doesn't hurt at all, there's a lot more wrong to fix first). Update your model of reality, avoid availability and (dis)confirmation bias. Face reality head-on. That kind of thing.

2MugaSofer12y

All due respect, but I don't think there is any imperative to view material that will cause serious psychological harm. If the price of knowledge is your right hand you are not required to pay it. The most relevant example would be showing hardcore pornography to small children - it may be knowledge of a kind, but it does more harm than good. EDIT: Of course, I read them anyway. Very, ah ... deep.

2Multiheaded12y

I only said that in regards to the many photos of staged degradation, multilation, self-harm, etc on the second blog - they're cringe-inducing yet presumably taken in a consensual way (for a loose definition of consent), and thus have a worse "unpleasantness-to-facing-reality" ratio than the textual descriptions of real abuse & its consequences. I quite agree that reading those shocking descriptions is epistemologically and morally imperative.

0DaFranker12y

Good point. If there are written descriptions available, they're a better weighted alternative.

1TheOtherDave12y

It would surprise me if they did, given that Omelas was constructed as an intuition pump.

5CCC13y

Omelas is a cognitive exploit, yes. That's really the point - it forces people to consider how appropriate their heuristics really are. Some people would make Omelas if they could; some wouldn't, for the sake of the one child. A firm preference for either possibility can be controversial, partially because there are good reasons for both states and partially because different ethical heuristics get levered in different directions. (A heuristic that compares the number of people helped vs. the number hurt will pull one way; a heuristic that says "no torture" will pull the other way).

3Desrtopa13y

Personally, on reading the story, I decided immediately that not only would I not walk away from Omelas (which solves nothing anyway,) I was fully in favor of the building of Omelases, provided that even more efficient methods of producing prosperity were not forthcoming. The prevention of dust specks may vanish into nothing in my intuitive utility calculations, but it immediately hits me that a single tortured child is peanuts besides the cumulative mass of suffering that goes on in our world all the time. With a hundred thousand dollars or so to the right charity, you could probably prevent a lot more disutility than that of the tortured child. If for that money I could either save one child from that fate, or create a city like Omelas minus the sacrifice, then it seems obvious to me that creating the city is a better bargain.

6Ghatanathoah13y

That's true, but I think that human values are so complex that any attempt to compress morality into one sentence is pretty much obligated to be vague. One rather obvious rejoinder is that there are currently hundreds, if not thousands of children who are in the same state as the unfortunate Omelasian right now in real life, so reducing the number to just one child would be a huge improvement. But you are right that even one seems too many. A more robust possibility might be to add "equality" to the list of the "good things in life." If you do that then Omelas might be morally suboptimal because the vast inequality between the child and the rest of the inhabitants might overwhelm the achievement of the other positive values. Now, valuing equality for its own sake might add other problems, but these could probably be avoided if you were sufficiently precise and rigorous in defining equality. I think the best explanation I've seem is something like the metaethics Eliezer espouses, which is (if I understand them correctly), that morality is a series of internally consistent concepts related to achieving what I called "the goods things in life," and that human beings (those who are not sociopaths anyway) care a lot about these concepts of wellbeing and want to follow and fulfill them. In other words, morality is like mathematics in some ways, it generates consistent answers(on the topic of people's wellbeing) that are objectively correct. But it is not like the Anti-Life Equation because it is not intrinsically motivating. Humans care about morality because of our consciences and our positive emotions, not because it is universally compelling. To put it another way, I think that if you were to give a superintelligent paperclipper a detailed description of human moral concepts and offered to help it make some more paperclips if it elucidated these concepts for you, that it would probably generate a lot of morally correct answers. It would feel no motivation to obey

0CCC13y

Vague or flawed. Given those options, I think I'd prefer vague. I agree completely. If I had any idea how Omelas worked, I might be tempted to try seeing if any of those ideas could be used to improve current societies. Hmmm. To avoid Omelas, equality would have to be fairly heavily weighted; any finite weighting given to equality, however, will simply mean that Omelas is only possible given a sufficiently large population (by balancing the cost of the inequality with the extra happiness of the extra inhabitants). Personally, I think that valuing equality itself is a good idea, if mixed in with a suitable set of other values; one possible failure mode for overvaluing equality is an equality of wretchedness, a state of "we're all equal because we all have nothing and no hope" (this is counteracted by providing suitable weights for the other good things, like happiness and the freedom to try to achieve goals (but what about the goal of vengeance? For a real slight? An imagined slight?)). An example of a society failing due to an over-reliance on equality, is France shortly after the French Revolution. I think that I should read through that entire sequence in the near future. I'd like to see his take on metaethics. Huh. I think we're defining 'morality' slightly differently here. My definition of 'morality' would be 'a set of rules, decided by some system, such that one can feed in a given action and (usually) get out whether that action was a good or a bad action'. Implicit in that definition is the idea that two people may disagree on what those rules actually are - that there might be better or worse moralities, and that therefore the answers given by a randomly chosen morality need not be objectively correct. To take an example; certain ancient cultures may have had the belief that human sacrifice was necessary, on Midwinter's Day, to persuade summer to come back and let the crops grow. In such a culture, strapping someone down and killing them in a partic

8Ghatanathoah13y

I think a possible solution would be to have equality and the other values have diminishing returns relative to each other. So in a society with a lot of other good things there is a great obligation to increase equality, whereas in a society with lots of suffering people it's more important to do whatever it takes raise the general level of happiness and not worry as much about equality. So a place as wondrous as Omelas would have a great obligation to help the child. I think one possible way to frame equality to avoid this is to imagine, metaphorically, that positive things give a society "morality points" and negative things give it "negative morality points." Then have it so that a positive deed that also decreases equality gets "extra points," while a negative deed that also exasperates inequality gets "extra negative points." So in other words, helping the rich isn't bad, it's just much less good than helping the poor. This also avoids another failure mode: Imagine an action that hurts every single person in the world, and hurts the rich 10 times as much as it hurts the poor. Such an action would increase equality, but praising it seems insane. Under the system I proposed such an action would still count as "bad," though it would be a bit less bad than a bad action that also increased inequality. I don't think that's that different from what I'm saying, I may be explaining it poorly. I do think that morality is essentially like a set of rules or an equation that one uses to evaluate actions. And I consider it objective in that the same equation should produce the same result each time an identical action is fed into it, regardless of what entity is doing the feeding. Then it is up to our moral emotions to motivate us to take actions the equation would label as "good." Describing it like that sounds a bit clinical though, so I'd like to emphasize that moral rules and equations, are ultimately about people's wellbeing and increasing the good things in life.

0CCC13y

That seems to work very well. So the ethical weight of a factor can be proportional to the reciprocal thereof (perhaps with a sign change). Then, for any number of people, there is a maximum happiness-factor that the equation can produce. So. This can be used to make an equation that makes Omelas bad for any sized population. But not everyone agrees that Omelas is bad in the first place; so is that necessarily an improvement to your ethical equation? That failure mode can also be dealt with by combining equality with other factors, such as not being hurt. (The relative weightings assigned to these factors would be important, of course). That seems like a reasonable definition; my point is that not everyone uses the same equation. Hmmm. You're right - that was a bad example. (I don't know if you're familiar with the Chanur series, by C. J. Cherryh? I ask because my first thought for a better example came straight out of there - she does a god job of presenting alien moralities) Let me provide a better one. Consider Marvin, and Fred. Marvin's moral system considers the total benefit to the world of every action; but he tends to weight actions in favour of himself, because he knows that in the future, he will always choose to do the right thing (by his morality) and thus deserves ties broken in his favour. Fred's moral system entirely discounts any benefits to himself. He knows that most people are biased to themselves, and does this in an attempt to reduce the bias (he goes so far as to be biased in the opposite direction). Both of them get into a war. Both end up in the following situation: Trapped in a bunker, together with one allied soldier (a stranger, but on the same side). An enemy manages to throw a grenade in. The grenade will kill both of them, unless someone leaps on top of it, in which case it will only kill that one. Fred leaps on top of the grenade. His morality values the life of the stranger over his own, and he thus acts to save the stranger

0Ghatanathoah13y

That's true, the question is, how often is this because people have totally different values, and how often is it that they have extremely similar "ideal equations," but different "approximations" of what they think that equation is. I think for sociopaths, and other people with harmful ego-syntonic mental disorders it's probably the former, but its more often the later for normal people. Eliezer has argued that it is confusing and misleading to use the word "morality" to refer to codes of behavior entities possess that have nothing to do with improving people's wellbeing, making the world a happier, fairer, freer place, and similar concepts. He argues that creatures like the Pebblesorters do not care about morality at all, they care about sorting pebbles and calling sorting pebbles a type of "morality" confuses two separate concepts. It sounds to me like Fred and Marvin both care about achieving similar moral objectives, but have different ideas about how to go about it. I'd say that again, which moral code is better can only be determined by trying to figure out which one actually does a better job of achieving moral goals. "Moral progress" can be regarded as finding better and better heuristics to achieve those moral goals, and finding a closer representation of the ideal equation. Again, I think I agree with Eliezer that a truly alien code of behavior, like that exhibited by sociopaths, and really inhuman aliens like the Pebblesorters or paperclippers, should maybe be referred to by some word other than morality. This is because since the word "morality" usually refers to doing things like making the world a happier place and increasing the positive things in life. So if we refer to the behavior code of a creature that cares nothing for doing those things as "morality," we will give the subconscious impression that that creature really does care about doing good and simply disagrees about how to go about it. This isn't correct, a sociopaths and paperclippers

2CCC13y

I'd say sometimes A, and sometimes B. But I think that's true even in the absence of mental disorders; I don't think that the "ideal equation" necessarily sits somewhere hidden in the human psyche. That is valid, as long as both systems have the same goals. Marvin's system includes the explicit goal "stay alive", more heavily weighted then the goal "keep a stranger alive"; Fred's system explicitly entirely excludes the goal "stay alive". If two moral systems agree both on the goals to be achieved, and the weightings to give those goals, then they will be the same moral system, yes. But two people's moral systems need not agree on the underlying goals. Well, to be fair, in a Paperclipper's mind, paperclips are the positive things in life, and they certainly make the paperclipper happier. I realise that's probably not what you intended, but the phrasing may need work. Which really feeds into the question of what goals a moral system should have. To the Babyeaters, a moral system should have the goal of eating babies, and they can provide a lot of argument to support that point - in terms of improved evolutionary fitness, for example. I think that we can agree that a moral system's goals should be the good things in life. I'm less certain that we can agree on what those good things necessarily are, or on how they should be combined relative to each other. (I expect that if we really go to the point of thoroughly dissecting what we consider to be the good things in life, then we'll agree more than we disagree; I expect we'll be over 95% in agreement, but not quite 100%. This is what I generally expect for any stranger). For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority. Hmmm. I haven't read that one.

1Ghatanathoah13y

It's not that I think there's literally a math equation locked in the human psyche that encodes morality. It's more like there are multiple (sometimes conflicting) moral values and methods for resolving conflicts between them and that the sum of these can be modeled as a large and complicated equation. You gave me the impression that Marvin valued "staying alive" less as an end in itself, and more as a means to achieve the end of improving the world. in particular when you said this: This is actually something that bothers me in fiction when a character who is superhumanly good and power (i.e. Superman, the Doctor) risks their lives to save a relatively small amount of people. It seems short-sighted of them to do that since they regularly save much larger groups of people and anticipate continuing to do so in the future, so it seems like they should preserve their lives for those people's sakes. If you define "the good things in life" as "whatever an entity wants the most," then you can agree, whatever someone wants is "good," be it paperclips or eudaemonia. On the other hand, I'm not sure we should do this, there are some hypothetical entities I can imagine where I can't see it as ever being good that they get what they want. For instance I can imagine a Human-Torture-Maximizer that wants to do nothing but torture human beings. It seems to me that even if there were a trillion Human-Torture-Maximizers and one human in the universe it would be bad for them to get what they want. For more neutral, but still alien preferences, I'm less sure. It seems to me that I have a right to stop Human-Torture-Maximizers from getting what they want. But would I have the right to stop paperclippers? Making the same paperclip over and over again seems like a pointless activity to me, but if the paperclippers are willing to share part of the universe with existing humans do I have a right to stop them? I don't know, and I don't think Eliezer does either. I think that we, and mos

0CCC12y

I'm sorry, there's an ambiguity there - when you say "the sum of these", are you summing across the moral values and imperatives of a single person, or of humanity as a whole? You are quite correct. I apologise; I changed that example several times from where I started, and it seems that one of my last-minute changes actually made it a worse example (my aim was to try to show how the explicit aim of self-preservation could be a reasonable moral aim, but in the process I made it not a moral aim at all). I should watch out for that in the future. I've always felt that was because one of the effects of great power, is that it's so very easy to let everyone die. With great power, as Spiderman is told, comes great responsibility; one way to ensure that you're not letting your own power go to your head, is by refusing to not-rescue anyone. After all, if the average science hero lets everyone he thinks is an idiot die, then who would be left? Sometimes there's a different reason, though; Sherlock Holmes would ignore a straightforward and safe case to catch a serial killer in order to concentrate on a tricky and deadly case involving a stolen diamond; he wasn't in the detective business to help people, he was in the detective business in order to be challenged, and he would regularly refuse to take cases that did not challenge him. (That's probably a fair example as well, actually; for Holmes, only the challenge, the mental stimulation of a worthy foe, is important; for Superman, what is important is the saving of lives, whether from a mindless tsunami or Lex Luthor's latest plot). Hmmm. If you're willing to accept zero, or near-zero, as a priority, then that statement can apply to any two sets of desires. Consider Sherlock holmes and a paperclipper; Holmes' desire for mental stimulation is high-priority, his desire for paperclips is zero-priority, while the paperclipper's desire for paperclips is high priority, and its desire for mental stimulation is zero-priority. (

0Ghatanathoah12y

Nearly all of humanity as a whole. There are obviously some humans who don't really value morality, we call them sociopaths, but I think most humans care about very similar moral concepts. The fact that people have somewhat different personal preferences and desires at first might seem to challenge this idea, but I don't really think it does. It just means that there are some desires that generate the same "value" of "good" when fed into the "equation." In fact, if diversity is a good, as we discussed previously, then people having different personal preferences might in fact be morally desirable. That's a good point. I was considering using the word "proportionality" instead of "priority" to better delineate that I don't accept zero as a priority, but rejected it because it sounded clunky. Maybe I shouldn't have. I agree with that. What I'm wondering is, would I have a moral duty to share resources with a paperclipper if it existed, or would pretty much any of the things I spend the resources on if I kept them for myself count (i.e. eudaemonic things) as "something more important." I think there might actually be lots of people like this, but most appear normal because they place even greater negative value on doing something stupid because they ignored good advice just because it came from an authority. In other words, following authority is a negative terminal value, but an extremely positive instrumental value. Exactly. I would still want the world to be full of a diverse variety of people, even if I had a nonsentient AI that was right about everything and could serve my every bodily need.

0CCC12y

Okay then, next question; how do you decide which people to exclude? You say that you are excluding sociopaths, and I think that they should be excluded; but on exactly what basis? If you're excluding them simply because they fail to have the same moral imperatives as the ones that you think are important, then that sounds very much like a No True Scotsman argument to me. (I exclude them mainly on an argument of appeal to authority, myself, but that also has logic problems; in either case, it's a matter of first sketching out what the moral imperative should be, then throwing out the people who don't match). And for a follow-up question; is it necessary to limit it to humanity? Let us assume that, ten years from now, a flying saucer lands in the middle of Durban, and we meet a sentient alien form of life. Would it be necessary to include their moral preferences in the equation as well? Even if they are Pebblesorters? It may be, but only within a limited range. A serial killer is well outside that range, even if he believes that he is doing good by only killing "evil" people (for some definition of "evil"). Hmmm. I think I'd put "buying a packet of paperclips for the paperclipper" as on the same moral footing, more or less, as "buying an icecream for a small child". It's nice for the person (or paperclipper) recieving the gift, and that makes it a minor moral positive by increasing happiness by a tiny fraction. But if you could otherwise spend that money on something that would save a life, then that clearly takes priority. Hmmm. Good point; that is quite possible. (Given how many people seem to follow any reasonably persuasive authority, though, I suspect that most people have a positive priority for this goal - this is probably because, for a lot of human history, peasants who disagreed with the aristocracy tended to have fewer descendants unless they all disagreed and wiped out said aristocracy). Here's a tricky question - what exactly are the limits of "non

3Ghatanathoah12y

I don't think I'm coming across right. I'm not saying that morality is some sort of collective agreement of people in regards to their various preferences. I'm saying that morality is a series of concepts such as fairness, happiness, freedom etc., that these concepts are objective in the sense that it can be objectively determined how much fairness, freedom, happiness etc. there is in the world, and that the sum of these concepts can be expressed as a large equation. People vary in their preference for morality, most people care about fairness, freedom, happiness, etc. to some extent. But there are some people who don't care about morality at all, such as sociopaths. Morality isn't a preference. It isn't the part of a person's brain that says "This society is fair and free and happy, therefore I prefer it." Morality is those disembodied concepts of freedom, fairness, happiness, etc. So if a person doesn't care about those things, it doesn't mean that freedom, fairness, happiness, etc. aren't part of their morality. It means that person doesn't care about morality, they care about something else." To use the Pebblesorter analogy again, the fact that you and I don't care about sorting pebbles into prime-numbered heaps isn't because we have our own concept of "primeness" that doesn't include 2, 3, 5 and 7. It just means we don't care about primeness. To make another analogy, if most people preferred wearing wool clothes but one person preferred cotton, that wouldn't mean that that person had their own version of wool, which was cotton. It means that that person doesn't prefer wool. Look inward, and consider why you think most people should be included. Presumably it's because you really care a lot about being fair. But that necessarily means that you cared about fairness before you even considered what other people might think. Otherwise it wouldn't have even occurred to you to think about what they preferred in the first place. The fact that most humans care, to

1TheOtherDave12y

So, OK. Suppose, on this account, that you and I both care about morality to the same degree... that is, you don't care about morality more than I do, and I don't care about morality more than you do. (I'm not sure how we could ever know that this was the case, but just suppose hypothetically that it's true.) Suppose we're faced with a situation in which there are two choices we can make. Choice A causes a system to be more fair, but less free. Choice B leaves that system unchanged. Suppose, for simplicity's sake, that those are the only two choices available, and we both have all relevant information about the system. On your account, will we necessarily agree on which choice to make? Or is it possible, in that situation, that you might choose A and I choose B, or vice-versa?

2Ghatanathoah12y

I think it depends on the degree of the change. If the change is very lopsided (i.e -100 freedom, +1 fairness) I think we'd both choose B. If we assume that the degree of change is about the same (i.e. +1 fairness, -1 freedom) it would depend on how much freedom and fairness already exist. If the system is very fair, but very unfree, we'd both choose B, but if it's very free and very unfair we'd both choose A. However, if we are to assume that the gain in fairness and the loss in freedom are of approximately equivalent size and the current system has fairly large amounts of both freedom and fairness (which I think is what you meant) then it might be possible that we'd have a disagreement that couldn't be resolved with pure reasoning. This is called moral pluralism, the idea that there might be multiple moral values (such as freedom, fairness, and happiness) which are objectively correct, imperfectly commensurable with each other, and can be combined in different proportions that are of approximately equivalent objective moral value. If this is the case then your preference for one set of proportions over the other might be determined by arbitrary factors of your personality. This is not the same as moral relativism, as these moral values are all objectively good, and any society that severely lacks one of them is objectively bad. It's just that there are certain combinations with different proportions of values that might be both "equally good," and personal preferences might be the "tiebreaker." To put it in more concrete terms, a social democracy with low economic regulation and a small welfare state might be "just as good" as a social democracy with slightly higher economic regulation and a slightly larger welfare state, and people might honestly and irresolvably disagree over which one is better. However, both of those societies would definitely be objectively better than Cambodia under the Khmer Rouge, and any rational, fully informed person who cares about

1TheOtherDave12y

I agree with you that in sufficiently extreme cases, we would both make the same choice. Call that set of cases S1. I think you're saying that if the case is not that extreme, we might not make the same choice, even though we both care equally about the thing you're using "morality" to refer to. I agree with that as well. Call that set of cases S2. I also agree that even in S2, there's a vast class of options that we'd both agree are worse than either of our choices (as you illustrate with the Khmer Rouge), and a vast class of options that we'd both agree are better than either of our choices, supposing that we are as you suggest rational informed people who care about the thing you're using "morality" to refer to. If I'm understanding you, you're saying in S2 we are making different decisions, but our decisions are equally good. Further, you're saying that we might not know that our decisions are equally good. I might make choice A and think choice B is wrong, and you might make choice B and think choice A is wrong. Being rational and well-informed people we'd agree that both A and B are better than the Khmer Rouge, and we might even agree that they're both better than fighting over which one to adopt, but it might still remain true that I think B is wrong and you think A is wrong, even though neither of us thinks the other choice is as wrong as the Khmer Rouge, or fighting about it, or setting fire to the building, or various other wrong things we might choose to evaluate. Have I followed your position so far?

0Ghatanathoah12y

Yes, I think so.

0TheOtherDave12y

OK, good. It follows that if a choice can go one of three ways (c1, c2, c3) and if I think c1> c2 > c3 and therefore endorse c1, and if you think c2 > c1 > c3 and therefore endorse c2, and if we're both rational informed people who are in possession of the same set of facts about that choice and its consequences, and if we each think that the other is wrong to endorse the choice we endorse (while still agreeing that it's better than c3), that there are (at least) two possibilities. One possibility is that c1 and c2 are, objectively, equally good choices, but we each think the other is wrong anyway. In this case we both care about morality, even though we disagree about right action. Another possibility is that c1 and c2 are, objectively, not equally good. For example, perhaps c1 is objectively bad, violates morality, and I endorse it only because I don't actually care about morality. Of course, in this case I may use the label "morality" to describe what I care about, but that's at best confusing and at worst actively deceptive, because what I really care about isn't morality at all, but some other thing, like prime-numbered heaps or whatever. Yes? So, given that, I think my question is: how might I go about figuring out which possibility is the case?

0Ghatanathoah12y

I'd say it's misleading to say we thought the other person was "wrong," since in this context "wrong" is a word usually used to describe a situation where someone is in objective moral error. It might be better to say: "c1 and c2 are, objectively, equally morally good, but we each prefer a different one for arbitrary, non-moral reasons." This doesn't change your argument in any way, I just think it's good to have the language clear to avoid accidentally letting in any connotations that don't belong. This is not something I have done a lot of thinking on since the odds of ever encountering such a situation are quite low at the present. It seems to me, however, that if you are this fair to your opponent, and care this much about finding out the honest truth, that you probably care at least somewhat about morality. (This brings up an interesting question, which is: might there be some "semi-sociopathic" humans who care about morality, but incrementally, not categorically? That is, if one of these people was rational, fully informed, lacking in self deception, and lacked akrasia, they would devote maybe 70% of their time and effort to morality and 30% to other things? Such a person, if compelled to be honest, might admit that c2 is morally worse than c1, but they don't care because they've used up their 70% of moral effort for the day. It doesn't seem totally implausible that such people might exist, but maybe I'm missing something about how moral psychology works, maybe it doesn't work unless it's all or nothing.) As for determining if your opponent care about morality, you might look to see if they exhibit any of the signs of sociopathy. You might search their arguments for signs of anti-epistemology, or plain moral errors. If you don't notice any of these things, you might assign a higher probability to prediction that your disagreement is to preferring different forms of pluralism Of course, in real life perfectly informed, rational humans who lack self decepti

0TheOtherDave12y

OK. In which case I can also phrase my question as, when I choose c1 over c2, how can I tell whether I'm making that choice for objective moral reasons, as opposed to making that choice for arbitrary non-moral reasons? You're right that it doesn't really change the argument, I'm just trying to establish some common language so we can communicate clearly. For my own part, I agree with you that ignorance and akrasia are major influences, and I also believe that what you describe as "incremental caring about morality" is pretty common (though I would describe it as individual values differing).

0Ghatanathoah12y

Wikipedia's page on internalism and externalism calls an entity that understands moral arguments, but is not motivated by them an "amoralist." We could say that a person who cares about morality incrementally to have individual values that are part moralist and part amoralist. It's hard to tell how many people are like this due to the confounding factors of irrationality and akrasia. But I think it's possible that there are some people who, if their irrationality and akrasia were cured, would not act perfectly moral. These people would say "I know that the world would be a better place if I acted differently, but I only care about the world to a limited extent." However, considering that these people would be rational and lack akrasia, they would still probably do more moral good than the average person does today.

0TheOtherDave12y

Would they say necessarily say that? Or might they instead say "I know you think the world would be a better place if I acted differently, but actually it seems to me the world is better if I do what I'm doing?"

0Ghatanathoah12y

That depends on if they are using the term "better" to mean "morally better" or "more effectively satisfies the sum total of all my values, both moral or non-moral." If the person is fully rational, lacking in self-deception, is being totally honest with you, and you and they had both agreed ahead of time that the word "better" means "morally better" then yes, I think they would say that. If they were lying, or they thought you were using the term "better" to mean "more effectively satisfies the sum total of all my values, both moral or non-moral," then they might not.

0TheOtherDave12y

I agree with you that IF (some of) my values are not moral and I choose to maximally implement my values, THEN I'm choosing not to act so as to make the world a morally better place, and IF I somehow knew that those values were not moral, THEN I would say as much if asked (supposing I was aware of my values and I was honest and so forth). But on your account I still don't see any way for me to ever know which of my values are moral and which ones aren't, no matter how self-aware, rational, or lacking in self-deception I might be. Also, even if I did somehow know that, and were honest and so forth, I don't think I would say "I only care about the world to a limited extent." By maximizing my values as implemented in the world, I would be increasing the value of the world, which is one way to express caring about the world. Rather, I would say "I only care about moral betterness to a limited extent; there are more valuable things for the world to be than morally better."

0Ghatanathoah12y

How does a Pebblesorter know it's piles are prime? The less intelligent and rational probably use some sort of vague intuition. The more intelligent and rational probably try dividing the numbers of pebbles by a number other than one. If you had full knowledge of the concept of "morality" and all the various sub-concepts it included, you could translate that concept into a mathematical equation (the one I've been discussing with CC lately), and see if the various values of yours that you feed into it return positive numbers. If your knowledge is more crude (i.e. if you're a real person who actually exists), then a possible way to do it would be to divide the nebulous super-concept of "morality" into a series of more concrete and clearly defined sub-concepts that compose it (i.e. freedom, happiness, fairness, etc.). It might also be helpful to make a list of sub-concepts that are definitely not part of morality (possible candidates include malice, sadism, anhedonia, and xenophobia). After doing that you could, if you are not self-deceived, use introspection to figure what you value. If you find that your values include the various moral sub-concepts, then it seems like you value morality. If you find yourself not valuing the moral sub-concepts, or valuing some nonmoral concept, then you do not value morality, or value non-moral things. As Eliezer puts it: The moral equation we are looking for isn't something that will provide us with a ghostly essence. It is something that will allow us to sum up and aggregate all the seperate good things like truth, happiness, and sentient life, so that we can effectively determine how good various combinations of these things are relative to each other, and reach an optimal combo. Do you want people to be happy, free, be treated fairly, etc? Then you value morality to some extent. Do you love torturing people just for the hell of it, or want to convert all the matter in the universe into paperclips? Then you, at the very leas

0TheOtherDave12y

I 100% agree that there is no ghostly essence of goodness. I agree that pursuing amoral, or even immoral, values can still lead to moral results. (And also vice-versa.) I agree that if I somehow knew what was moral and what wasn't, then I would have a basis for formally distinguishing my moral values from my non-moral values even when my intuitions failed. I could even, in principle, build an automated mechanism for judging things as moral or non-moral. (Similarly, if a Pebblesorter knew that primeness was what it valued and knew how to factor large numbers, it would have a basis for formally distinguishing piles it valued from piles it didn't value even when its intuitive judgments failed, and it could build an automated mechanism for distinguishing such piles.) I agree with you that real people who actually exist can't do this, at least not in detail. You suggest we can divide morality into subconcepts that comprise it (freedom, happiness, fairness, etc.) and that it excludes (anhedonia, etc.). What I continue to not get is, on your account, how I do that in such a way as to ensure that what I end up with is the objectively correct list of moral values, which on your account exists, rather than some different list of values. That is, suppose Sam and George both go through this exercise, and one of them ends up with "freedom" on their list but not "cooperation", and the other ends up with "cooperation" but not "freedom." On your account it seems clear that at least one of them is wrong, because the correct list of moral values is objective. So, OK... what would we expect to experience if Sam were right? How does that differ from what we would expect to experience if George were right, or if neither of them were? Again: how do we know that? What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does? What should I be looking for, to notice if this

0Ghatanathoah12y

Well, I am basically asserting that morality is some sort of objective equation, or "abstract idealized dynamic," as Eliezer calls it, concerned with people's wellbeing. And I am further asserting that most human beings care very much about this concept. I think this would make the following predictions: 1. In a situation where a given group of humans had similar levels of empirical knowledge and a similar sanity waterline there would be far more moral agreement among them than would be predicted by chance, and far less moral disagreement than is mentally possible. 2. It is physically possible to persuade people to change their moral values by reasoned argument. 3. Inhabitants of a society who are unusually rational and intelligent will be the first people in that society to make moral progress, as they will be better at extrapolating answers out of the "equation." 4. If one attempted to convert the moral computations people make into an abstract, idealized process, and determine it's results, many people would find those results at least somewhat persuasive, and may find their ethical views changed by observing them. All of these predictions appear to be true: 1. Human societies tend to have a rather high level of moral agreement between their members. Conformity is not necessarily an indication of rightness, it seems fairly obvious that whole societies have held gravely mistaken moral views, such as those that believed slavery was good. However, it is interesting that all those people in those societies were mistaken in exactly the same way. That seems like evidence that they were all reasoning towards similar conclusions, and the mistakes they made were caused by common environmental factors that impacted all of them. There are other theories that explain this data of course, (peer pressure, for instance), but I still find it striking. 2. I've had moral arguments made by other people change my mind, and changed the minds of other people by moral argu

2TheOtherDave12y

All of those predictions seem equally likely to me whether Sam is right or George is, so don't really engage with my question at all. At this point, after several trips 'round the mulberry bush, I conclude that this is not because I'm being unclear with my question but rather because you're choosing not to answer it, so I will stop trying to clarify the question further. If I map your predictions and observations to the closest analogues that make any sense to me at all, I basically agree with them. It is. It doesn't bother me; it's a fine thing to do under some circumstances. If we can agree that that's what we're doing when we talk about "objective morality," great. If not (which I find more likely), never mind. As above, I don't see what the word "moral" is adding to this sentence. But sure, unfairly claiming status bothers me to the extent that I care about fairness. (That said, I don't think claiming status by describing my values as "moral" is unfair; pretty much everybody has an equal ability to do it, and indeed they do. I just think it confuses any honest attempt at understanding what's really going on when we decide on what to do.) It depends on why and how I value ("care about") fairness. If I value it instrumentally (which I do), then it makes perfect sense to say that being fair to people who treat others unfairly is net-valuable, although it might be true or false in any given situation depending on what is achieved by the various kind of fairness that exist in tension in that situation. Similarly, if I value it in proportion to how much of it there is (which I do), then it makes sense to say that, although it might be true or false depending on how much fairness is gained or lost by doing so. (nods) Totally. And the ability to phrase ideas in silly-sounding ways is valuable for rhetorical purposes, although it isn't worth much as an analytical tool.

0Ghatanathoah12y

I'm really sorry, I was trying to kill two birds with one stone and simultaneously engage that question and your later question ["What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does?"] at the same time, and I ended up doing a crappy job of answering both of them. I'll try to just answer the Sam and George question now. I'll start by examining the Pebblesorters P-George and P-Sam. P-George thinks 9 is p-right and 16 is p-wrong. P-Sam thinks 9 is p-wrong and 16 is p-right. They both think they are using the word "p-right" to refer to the same abstract, idealized process. What can they do to see which one is right? 1. They assume that most other Pebblesorters care about the same abstract process they do, so they can try to persuade them and see how successful they are. Of course, even if all the Pebblesorters agree with one of them, that doesn't necessarily mean that one is p-correct, those sorters may be making the same mistake as P-George or P-Sam. But I think it's non-zero Bayesian evidence of the p-rightness of their views. 2. They can try to control for environmentally caused error by seeing if they can also persuade Pebblesorters who live in different environments and cultures. 3. They can find the most rational and p-sane Pebblesorting societies and see if they have an easier time persuading them. 4. They can actually try to extrapolate what the abstract, idealized equation that the word "p-right" represents is and compare it to their views. They read up on Pebblesorter philospher's theories of p-rightness and see how they correlate with their views. Pebblesorting is much simpler than morality, so we know that the abstract, idealized dynamic that the concept "p-right" represents is "primality." So we know that P-Sam and P-George are both partly right and partly wrong, 9 and 16 both aren't prime. Now let's translate that into human. We

0TheOtherDave12y

OK. Thanks for engaging with the question; that was very helpful. I now have a much better understanding of what you believe the differences-in-practice between moral and non-moral values are. Just to echo back what I'm hearing you say: to the extent that some set of values Vm is easier to convince humans to adopt than other sets of values and easier to convince sane, rational societies to adopt than less sane, less rational societies and better approximates the moral theories created by philosophers than other sets of values, to that extent we can be confident that Vm is the set of values that comprise morality. Did I get that right? Regarding fairmindedness: I endorse giving someone's advice consideration to the extent that I'm confident that considering their advice will implement my values. And, sure, it's unlikely that the advice of an unfairminded person would, if considered, implement the value of fairness.

0Ghatanathoah12y

Yes, all those things provide small bits of Bayesian evidence that Vm is closer to morality than some other set of values.

0CCC12y

It seems very likely that a person who cares a certain amount about morality, and a certain amount about money, would be willing to compromise his morality if given sufficient money. Such a mental model would form the basis of bribery. (It doesn't have to be money, either, but the principle remains the same). So a semi-sociopathic person would be anyone who could be bribed into completely disregarding morality.

1TheOtherDave12y

On this account, we could presumably also categorize a semi-semi-sociopathic person as one who could be bribed into partially disregarding the thing we're labeling "morality". And of course bribes needn't be money... people can be bribed by all kinds of things. Social status. Sex. Pleasant experiences. The promise of any or all of those things in the future. Which is to say, we could categorize a semi-semi-sociopath as someone who cares about some stuff, and makes choices consistent with maximizing the stuff they care about, where some of that stuff is what we're labeling "morality" and some of it isn't. We could also replace the term "semi-semi-sociopath" with the easier to pronounce and roughly equivalent term "person". It's also worth noting that there probably exists stuff that we would label "morality" in one context and "bribe" in another, were we inclined to use such labels.

0CCC12y

Ah, I think I see your point. What you're saying - and correct me if I'm wrong - is that there is some objective True Morality, some complex equation that, if applied to any possible situation, will tell you how moral a given act is. This is probably true. This equation isn't written into the human psyche; it exists independantly of what people think about morality. It just is. And even if we don't know exactly what the equation is, even if we can't work out the morality of a given act down to the tenth decimal place, we can still apply basic heuristics and arrive at a usable estimate in most situations. My question is, then - assuming the above is true, how do we find that equation? Does there exist some objective method whereby you, I, a Pebblesorter, and a Paperclipper can all independently arrive at the same definition for what is moral (given that the Pebblesorter and Paperclipper will almost certainly promptly ignore the result)? (I had thought that you were proposing that we find that equation by summing across the moral values and imperatives of humanity as a whole - excluding the psychopaths. This is why I asked about the exclusion, because it sounded a lot like writing down what you wanted at the end of the page and then going back and discarding the steps that wouldn't lead there; that is also why I asked about the aliens). Yes, I think we're in agreement on that. (Though this does suggest that 'sentient' may need a proper definition at some point).

1nshepperd12y

In the same way as there exists a True Set of Prime Numbers, and True Measure of How Many Paperclips There Are...

0Ghatanathoah12y

0CCC12y

Hmmm. That which evolution has "written" into the human psyche could, in theory, and given sufficient research, be read out again (and will almost certainly not be constant across most of humanity, but will rather exist with variations). But I doubt that morality is all in out genetic nature; I suspect that most of it is learned, from our parents, aunts, uncles, grandparents and other older relatives; I think, in short, that morality is memetic rather than genetic. Though evolution still happens in memetic systems just as well as in genetic systems. Hmmm. Looking at the wikipedia article, I can expect reflective equilibrium to produce a consistent moral framework. I also expect a correct moral framework to be consistent; but not all consistent moral frameworks are correct. (A paperclipper does not have what I'd consider a correct moral framework, but it does have a consistent one). If you start out close to a correct moral framework, then reflective equilibrium can move you closer, but it doesnt necessarily do so. Hmmm. The primary use of trying to find the True Morality Equation, to my mind, is to work it into a future AI. If we can find such an equation, prove it correct, and make an AI that maximises its output value, then that would be an optimally moral AI. This may or may not count as Friendly, but it's certainly a potential contender for the title of Friendly. Carrying through this method to completion could give us - or anyone else - an equation. But is there any way to be sure that it necessarily gives us the correct equation? (A pebblesorter may actually be a very good help in resolving this question; he does not care about morality, and therefore does not have any emotional investment in the research). The first thought that comes to my mind, is to have a very large group of researchers, divide them into N groups, and have each of these groups attempt, independently, to find an equation; if all of the groups find the same equation, this would be evid

0Ghatanathoah12y

That's possible. But memetics can't build morality out of nothing. At the very least, evolved genetics has to provide a "foundation," a part of the brain that moral memes can latch onto. Sociopaths lack that foundation, although the research is inconclusive as to what extent this is caused by genetics, and what extent it is caused by later developmental factors (it appears to be a mix of some sort). Yes, that's why I consider reflective equilibrium to be far from perfect. Depending on how many errors you latch onto, it might worsen your moral state. Considering how morally messed up the world is now, even an imperfect equation would likely be better (closer to being correct) than our current slapdash moral heuristics. At this point we haven't even achieved "good enough," so I don't think we should worry too much about being "perfect." That's not inconceivable. But I think that each of the subtly different equations would likely be morally better than pretty much every approximation we currently have.

0CCC12y

That sounds plausible, yes. Hmmm. Finding an approximation to the equation will probably be easier than step two; encouraging people worldwide to accept the approximation. (Especially since many people who do accept it will then promptly begin looking for loopholes; either to use or to patch them). However, if the correct equation cannot be found, then this means that the Morality Maximiser AI cannot be designed.

0Ghatanathoah12y

That's true, what I was trying to say is that a world ruled by a 99.99% Approximation of Morality Maximizer AI might well be far far better than our current one, even if it is imperfect. Of course, it might be a problem if we put the 99.99% Approximation of Morality Maximizer AI in power, then find the correct equation, only to discover that the 99AMMAI is unwilling to step down in favor of the Morality Maximizer AI. On the other hand, putting the 99AMM AI in power might be the only way to ensure a Paperclipper doesn't ascend to power before we find the correct equation and design the MMAI. I'm not sure whether we should risk it or not.

3TheOtherDave13y

Well, if we're really going to take Omelas seriously as our test case, then presumably we also have to look at how much that "extra happiness" (or whatever else we're putting in the plus column) is reduced by those who walk away from it, and by those who are traumatized by it, and so forth. It might turn out that increasing the population doesn't help. But that's just a quibble. I basically agree: once we swallow the assumption that for some reason we neither understand nor can ameliorate, the happiness of the many ineluctably depends on the misery of the few, then a total-utilitarian approach either says that equality is the most important factor in utility (which is a problem like you describe), or endorses the few being miserable. That's quite an assumption to swallow, though. I have no reason to believe it's true of the world I live in. A weaker version that might be true of the world I actually live in is that concentrating utility-generating resources in fewer hands results in higher total utility-from-all-sources-other-than-equality (Ua) but more total-disutility-from-inequality (Ub). But it's not quite as clear that our (Ua, Ub) preferences are lexicographic.

1CCC13y

Doubling the population should double the happiness; double the trauma; double the people who walk away. The end result should be (assuming a high enough population that the Law of Large Numbers is a reasonable heuristic) about twice the utility. Consider the case of farmland; larger farms produce more food-per-acre than smaller farms. (Why? Because larger farms attract commercial farmers with high-intensity farming techniques; and they can buy better farming equipment with their higher profits). Now, in the case of farmland, the optimal scenario is not equality; you don't want everyone to have the same amount of farmland, you want those who are good at farming to have most of it. (For a rather dramatic example of this, see the Zimbabwe farm invasions). On the other hand, consider the case of food itself. Here, equality is a lot more important; giving one man food for a hundred while ninety-nine men starve is clearly a failure case, as a lot of food ends up going rotten and ninety-nine people end up dead. So the optimal (Ua, Ub) ordering depends on exactly what it is that is being ordered; there is no universally correct ordering.

3TheOtherDave13y

You seem to be assuming a form of utility that is linear with happiness, with trauma, with food-per-acre, with starving people, etc. I agree with you that if we calculate utility this way, what you say follows. It's not clear to me that we ought to calculate utility this way.

0CCC13y

Hmmm. There are other ways to calculate utility, yes, and some of them are very likely better than linear. But all of them should at least be monotonically increasing with increased happiness, lower trauma, etc. There isn't some point of global happiness where you can say that global happiness above this level is worse than global happiness at this level, if all else remains constant. The increase may be smaller for higher starting levels of happiness, but it should be an increase. Such a system can either be bounded above by a maximum value, which it approaches asymptotically (such that no amount of global happiness alone can ever be worth, say, ten billion utilions, but it can approach arbitrarily close to that amount), or it can be unbounded (in which case enough global happiness can counter any finite amount of negative effects). A linear system would be unbounded, and my comments above can be trivially changed to fit with any unbounded system (but not necessarily with a bounded system). It's not clear to me whether it should be a bounded or an unbounded system..

3TheOtherDave13y

OK, so we agree that doubling the population doesn't provide twice the utility, but you're now arguing that it at least increases the utility (at least, up to a possible upper bound which might or might not exist). This depends on the assumption that the utility-increasing aspects of increased population increase with population faster than the utility-decreasing aspects of increased population do. Which they might not.

2CCC13y

...you know, it's only after I read this comment that I realised that you're suggesting that the utility-decreasing aspects may not use the same function as the utility-decreasing aspects. That is, what I was doing was mathematically equivalent to first linearly combining the separate aspects, and only then feeding that single number to a monotonically increasing nonlinear function. Now I feel somewhat silly. But yes, now I see that you are right. There are possible ethical models (example: bounded asymptotic increase for positive utility, unbounded linear decrease for negative utility) wherein a larger Omelas could be worse than a smaller Omelas, above some critical maximum size. In fact, there are some functions wherein an Omelas of size X could have positive utility, while an Omelas of size Y (with Y>X) could have negative utility.

0TheOtherDave13y

Yup. Sorry I wasn't clearer earlier; glad we've converged.

2nshepperd13y

If you're going to do that, why not just directly use happiness and fulfillment?

0CCC13y

I cannot create an entire ethical framework, for everyone to follow, on any basis, and expect that it will be able to hold up for the next thousand years. If I try, I will fail, and this is why: because people cheat. Many intelligent agents will poke at the rules, seeking a possible exploit thereof that enhances their success at the possible expense of their neighbours' success. Over the next thousand years, there will be thousands, probably millions, of such intelligent agents hunting for, and attempting to exploit, flaws in the system; people who stick by the letter of the rule, and avoid the spirit of the rule. I cannot create an entire ethical framework, because I cannot outwit thousands or millions of future peoples' attempts to find and exploit gaps and loopholes in my framework. Hence, the best that I can do is to find a system that has already endured a period of field testing and that hasn't broken yet; and perhaps attempt a small, incremental improvement (no more) in order to test that improvement.

0nshepperd13y

What does that have to do with the situation at hand? Morality is an abstract division of actions into right and wrong, not some set of laws laid down by philosophers on the rest of the population. If you're trying to work out what you mean by "morality" and use some criteria (such as something including happiness and fulfillment of populations which adopt that definition) to choose from a bunch of alternatives, then probably those criteria themselves are the most accurate definition of "morality" you could hope to find. I might add, in [almost] exactly the same way that a program which writes and then executes a program to add two numbers is, in fact, itself a program that adds two numbers. You can write out your final definition in legalese later, if the situation calls for it.

0CCC13y

Morality comes with an implicit rule; when it says that "this action is the right action to take in this situation", then the implicit rule is "if you find yourself in this situation, take this action". There is usually no Morality Policeman ready to administer punishment if the rule is not followed, and the choice to follow the rule or not remains; but the rule is there. The difficulty is that I know that the algorithm that I am following is very likely not to fulfil the criteria in the very best possible way; merely in (more or less) the best possible way that they have been fulfilled in the past. If I simply list the criteria, then I falsely imply that the chosen system of morality is the best fit for those criteria; and I am trying to avoid that implication.

0Peterdjones13y

Define it, or defend it? There are a lot of defences, but not so much definitions.

2Jayson_Virissimo13y

I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters' ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?

4wedrifid13y

Preferring other people like Larry to be homosexual is adaptive for me. And it is the judgement by others (and the implicit avoidance of that through shame) that we are considering here. That said: Absolutely, and the entire line of reasoning relies on conveying the speaker's own morality ("it is second-order 'right' to be homosexual") on others without making it explicit.

2MugaSofer13y

The same reason sorting pebbles into correct heaps was adaptive in the first place. EDIT: Wait, does it matter that homosexuality is probably not adaptive?

-2Jayson_Virissimo13y

That was the point of my comment. There is a large disanalogy between heterosexuality and 13-pebble heap preference (namely, the first highly adaptive, but the second has no apparent reason to be). Although, I'm not sure if that is enough to break the metaphor.

3MugaSofer13y

There are many properties homosexuality has but 11-pebble heap preference don't, and vice versa. Why is evolutionary maladaptiveness worth pointing out, is my question.

0Jayson_Virissimo13y

Well, if moral norms are the Nash equilibria that result from actual historical bargaining situations (that are determined largely by human nature and the ancestral environment), then it seems somewhat relevant. If moral norms are actually imperative sentences uttered by God, then it seems completely irrelevant. Etc... I suppose whether or not the pebble-sorting metaphor is good depends on which meta-ethical theory is true. In other words, I'm agreeing with PhilGoetz; Example 2 and Example 3 are only in separate classes of meta-wants assuming a (far from universally shared) moral system.

2Ghatanathoah13y

I would regard moral norms as useful heuristics for achieving morally good results, not as morality in and of itself. I think that some sort of ethical naturalism (or "moral cognitivism" as Eliezer calls it) is correct, where "morally good" is somewhat synonymous with "helps people live lives full of positive values like love, joy, freedom, fairness, high challenge, etc." There is still much I'm not sure of, but I think, that is probably pretty close to the meaning of right. In Larry's case I would argue that homosexual relationships usually do help people live such lives.

0MugaSofer13y

Oh, you mean that humans might genuinely dislike homosexuality as a terminal value, because evo-psych. ... huh.

0MugaSofer13y

Incidentally, it's easier to sort pebbles into heaps of 11. The original pebblesorters valued larger heaps, but had a harder time determining their correctness.

3Ghatanathoah13y

That's why I was careful to refer to them as 11-Pebble and 13-Pebble Favorers. They do value other sizes of pebble heaps, 11 and 13 are just the numbers they do most frequently. Or perhaps 11 and 13 are the heaps they like making in their personal time, but they like larger prime numbers for social pebble-sorting endeavors. The point is, I said they "favored" that size because I wanted to make sure that the ease of sorting the piles didn't seem too relevant, since that would distract away from the central metaphor.

0MugaSofer13y

Oops.

2orthonormal16y

I suspect we're doing some extrapolation here in order to distinguish these cases. I expect that if Mimi knew more about herself and the world, and thought more clearly, she would still want to not want heroin; while I expect that if Larry knew more about himself and the world, and thought more clearly, he would be likely to reject the system of belief that causes him to think homosexuality immoral.

0mitechka16y

Alternatively, after sobering up, Mimi might decide that experiencing heroine high makes her life so much more fulfilling, that the much shortened life expectancy of a heroine addict doesn't seems to be a fair price to pay for it. As usual it is all up to personal definition of utility.

0Mike Bishop16y

Contra Cyan & Alicorn, I am inclined to go with PhilGoetz and "punt it off to your moral system, or your expected-value computations." Trying to change your homosexual desires will probably fail and create a lot of collatoral damage. I would guess that trying to change your desire for heroin is somewhat more likely to succeed, though I'm willing to consider the argument that heroin addicts should accept their addiction but attempt to minimize its harmful side effects.

0Cyan16y

I think the distinction is that we think of Mimi as wishing to revoke a decision made of her own free will; not so with Larry.

0[anonymous]16y

Mimi grew up in a society that taught her that heroin is bad. Larry grew up in a society that taught him that homosexuality is bad. How do you tell the difference from their point of view?

0Alicorn16y

I think it probably has something to do with the fact that Mimi (probably) wasn't born addicted to heroin (and even if she was, we can point to the behavior that caused it), whereas the consensus seems to be that homosexuality is innate.

1PhilGoetz16y

There must be more to it than that. If Larry were innately born attracted to children rather than to men, we probably wouldn't say it was okay.

1Alicorn16y

It's not about whether it's okay, it's about whether it's "part of who he is" or an alien intrusion.

4newerspeak16y

Okay. I'm Paul Erdos. I've been taking amphetamine and ritalin for 20-odd years to enhance my cognitive performance. In general I want to want these drugs, because they help me do good, important and enjoyable work, which is impossible for me without them. I can stop wanting these drugs when I want to, like when my friend bet me $500 that I couldn't. I wanted to win that bet, so I wanted not to want the drugs, so I stopped wanting them. Was that my only motivation? Also, I don't want others to want to want amphetamines just because I want to want amphetamines. A while ago I took Euler's place as the most prolific mathematician of all time.

2ABranco16y

Paul Erdös did it regularly, yes. Successfully, it seems — but I wonder about the costs. Does anyone have consistent data on that? Picking only Erdös' case, would, I'm afraid, be a case of both survivorship bias and hasty generalization.

3Richard_Kennaway16y

That doesn't solve PhilGoetz's example though. And in the original version of Larry, his parents might very well say that his revulsion at homosexual acts is "who he is" and his sexual feelings the "alien intrusion". Are these concepts anything but a way of making disguised moral judgements? Is "who someone really is" just "who I would prefer them to be"? Then again, another attitude to Larry is that his sexual feelings are who he really is, but that resisting them is a cross he has to bear. (I believe this is the Roman Catholic view.) So I don't think the concept of authenticity solves these problems.

[-]AnlamK16y40

Harry Frankfurt, who came up with the original idea, did a much better job in explaining in my opinion. (Why are you not referring to his paper?)

Here is the link for the curious: http://www.usfca.edu/philosophy/pdf%20files/Freedom%20of%20the%20Will%20and%20the%20Concept%20of%20a%20Person.pdf

1Alicorn16y

I probably should have mentioned Frankfurt's work, but I was being petty and declined to do so because he irritates me by calling second-order desire a criterion for personhood. Moreover, I wasn't trying to get into the notion of "will" or what second-order desire is for; I just wanted to provide a summary and some examples because someone had asked about it, and if the post is well-received I'll follow up with more complicated stuff.

1thomblake16y

Still, at least a hat-tip is obviously warranted. Hardly an excuse for academic dishonesty. Okay, this forum is hardly 'academic', but the point stands.

[-]Jordan16y30

It's not always so easy to say which desire is actually first order and which is second order.

For instance, example 3 could be inverted:

Larry was brought up to believe God hates homosexuality. Because of this he experiences genuine disgust when he thinks about homosexual acts, and so desires not to perform them or even think about them (first order). However, he really likes his friend Ted and sometimes wished God wasn't such a dick (second order).

There's likely even a third order desire: Larry was brought up to be a good Christian, and desperately wishes... (read more)

1Psychohistorian16y

This is not second order. It's just D(God approved of homosexuality). If he himself were God, then it would probably be second order, but just wanting the rules to be different is first-order. Similarly, is not third order. Third order gets weird. Third order would be that Mimi wants heroin, and in fact, wants to want heroin, (if you offered to magically make her not like heroin, she'd emphatically decline), but on top of that, she wants to want to not want to use heroin. Maybe she doesn't have a problem with it, but her friends do. Maybe she would be well served if she could express an honest desire to quit while not actually quitting. Third-order desires get pretty confusing, though I may have just explained this one poorly. Furthermore is not exactly in line with the post. I don't think there's any claim that these things are a logical hierarchy. One can have extremely weak first order desires and extremely strong second order desires, though the latter will tend to consolidate into changed first order desires if they are strong enough. Second order desires do often stem from conflicting first order desires, but what determines the order is the object of desire (if it itself is about a desire, it's 2nd order, and so on), not its magnitude.

1Jordan16y

I can see that I worded things in a misleading fashion. The main point is just that first orders desires and second order desires referring to them can often switch places. Rephrasing: Larry wants to be straight... (first order) ...but wants not to want to be straight so he can be with his friend Ted. (second order) By logical hierarchy I just mean the notion there is some well ordering of terms here. Each desire can be the first order desire to the other, forming a loop rather than a tier.

6pjeby16y

Actually, it's simpler to treat all desires as independent motivations. Whether they are "first-order" or "second-order" is a function of which is the organism's current goal. When an addict feels bad, their current goal is to not feel bad - so they indulge. Anything that would stop them from doing so -- including any desire to reform -- is now opposed, whether the thing being opposed is an external event or an internal desire to quit. Conversely, when the addict is satisfied, their current goal may be to feel better about themselves, at which point whatever obstacles to that goal become relevant, whether external events or internal desires. IOW, it's not the case that desires are ever "first order" or "second order" in and of themselves. So-called "second order" desires are merely subgoals that arise as a side-effect of a conflict between an active goal and another desire that opposes the goal in some way. And it's important to understand their contextual nature. A subgoal that's consistently reinforced can become a seemingly-independent desire via the standard "cached thought" or "promoted subgoal" mechanism, but this doesn't always occur. And until it occurs, the second-order desires are just temporary subgoals. See for example, Alicorn's example 1: the goal of liking Mountain Dew (or not disliking it) is strictly in the context of the alertness goal. If the alertness goal were satisfied in some other way, the subgoal would no longer be active.

0MichaelVassar16y

I'm pretty sure that you are pointing to a correct proposition here but have overstated your case. At the very least, separate from goals its possible to build non-desire habits which desires can affirm or conflict with. More importantly, you are missing the point that second order desires can be about first order desires without the first order desires, even when active, being about the second order desires.

1pjeby16y

How so? That is, how am I missing that point? I'm simply saying that second-order desire can only arise as a subgoal of some other desire, even if that other desire is to simply have a certain social image. This doesn't imply any sort of symmetry being required, so I'm not clear on why you think I said it does. (In fact, my reference to example 1 describes an asymmetric conflict case.)

0stcredzero16y

Asymmetry between 2nd order and 1st order desires can be explained easily if 2nd order desires didn't really exist, or if only one specific 2nd order desire existed, namely a desire for resolution. "I want to want X" then just becomes some person's rationalization about their conflicted situation. I find this idea attractive, because a desire for resolution seems a natural thing for a conscious being to have.

1pjeby16y

But such a thing isn't intrinsic. People routinely do things that are in conflict, without ever resolving the conflict. If anything, we have a drive to appear consistent to other people -- a more evolutionarily-relevant drive than a desire to actually be consistent. Meanwhile, second-order desires are just subgoals, like "walk across the room" is a subgoal of "get a glass of water". We experience wanting to (not) want something because it supports some other goal -- whether the other goal is something we want to admit to or not. But that other goal is never really "get some resolution" -- that's just a verbal explanation that deflects attention from whatever the real goal is. (Because without some conflicting goal being present, there would be nothing to "resolve"!)

0stcredzero16y

I disagree. Conflict resolution is intrinsic, however most people resolve many of their conflicts in irrational ways, including distraction. Come to think of it, a desire/goal-conflict resolution urge explains procrastination quite handily. Let's not confuse "a desire to resolve conflicting urges" with "a desire to be rationally self-consistent." These are two different things. Everyone will have the former. Some will be able to cultivate the latter. A drive to appear consistent to others is yet a third thing. I suspect that our drive for "getting resolution" is much like our preference for clearly annunciated speech, sunny vistas, and uncluttered rooms. We are driven to optimize our perception, and this drive is expressed as aesthetic desire. Our sense of the aesthetic even extends to internal perception of our ideas -- there is an attraction to elegant ideas. Religions often exploit this. For example, Islam is said to be popular in parts of the world because it presents itself as straightforward. I think you're right that "getting resolution" is not a goal. It is more like a drive. Much like our desire to see all of what we are observing often results in our craning our neck. Like other drives it can result in goals. I also like your subgoal formulation. I would posit that the urge towards "clarity" is what drives it. But remember, just because one can imagine some scenario and take it as a desired goal, doesn't mean that the situation is sensible. I think wanting to want X is along the same lines as wanting to hear the sound of one hand clapping. So, in the Mountain Dew example, the subject wants to stay awake and the subject also wants to avoid the unpleasant stimuli of Mountain Dew. To resolve this goal conflict, they formulate the subgoal, "I want to want Mountain Dew," which is a condition where there is no conflict. I note, however, that the subject wouldn't mind drinking chilled but flat Mountain Dew if it were readily available. Most likely they would

1pjeby16y

My point is that "a desire to resolve conflicting urges" is an unnecessary hypothesis. Conflict resolution is an emergent property of goal-seeking, not an independent goal or desire of itself, nor even a component of goal-seeking. If you have a goal to get a soda from the fridge, and therefore a subgoal of walking across the room, but there is something in your way, then you will desire to go around it. To posit even a "drive" to "get resolution" is adding unnecessary entities to the equation. Now, if you said that we experience conflict as painful, and desire to avoid it, I'd agree with you. However, experiencing the pain of conflict does not consistently motivate people to resolve the conflict. In fact, it frequently motivates people to avoid the subject entirely, so as to remove awareness of the conflict! That's why I believe that talking about "conflict resolution as intrinsic" or an urge to "get resolution" is both unnecessary and erroneous: people DO experience negative reinforcement from conflict, but this is not the same thing as a desire for resolution. In humans (as in all animals that I know of), a drive to avoid one thing does not produce the same results as a drive to approach its opposite (nor vice versa).

0stcredzero16y

A very good point. It's much more accurate to say that people have an aversion to internal conflicts, and that this is part of the inbuilt mechanism for mediating between conflicting desires. This is a better way to word what I am getting at. "Desire for resolution" can be easily misinterpreted. For example, I did not mean a "desire for a rational resolution in actuality." That would preclude the mechanism from being a factor in procrastination, and I believe it is a part of that. I think it is also related to the Paradox of Choice. http://www.amazon.com/Paradox-Choice-Why-More-Less/dp/0060005688 As with many evolved mechanisms, it works imperfectly, but well enough (especially when viewed in the context of a Stone Age denizen's life). I don't think it's an unnecessary entity, merely a mis-stated one. Remember the context of the OP. I thought we were talking about perceived conundrums. When a way to "go around it" is not immediately obvious, then one sometimes makes up an impractical subgoal, like "I want to want to have sex with women," from example 3. It's often the case that there is an awareness of the impracticality of such subgoals, and as such it offers inadequate relief, but the impractical subgoal still becomes a fixation. I have generally found myself in conundrums actively seeking some answer. But it seems reasonable that this is not going to be everyone's reaction, and the drive is actually avoidance of conflict.

0Alicorn16y

This is kind of tangential to your actual statement, but I've never found an originally carbonated beverage that was flat enough to be drinkable. I'd think it had more to do with the flavorings used in sodas if I didn't have the same problem with seltzer water and sparkling juices.

1stcredzero16y

But if someone had a magic powder that was tasteless and could remove all carbonation from a drink, then perhaps you could drink it and in a given context would want to. My point is that "2nd order desires" are probably just due to mis-formulated goals and subgoals. I don't think people really want to want X. Most often, they want X but don't also want Y, or they want X but cannot give up Y. I suspect it often helps if you can get as close to the level of basic drives as possible. In the Mountain Dew conundrum, it's self preservation and avoidance of noxious stimuli. These desires are not in conflict, only the particular goal+subgoal scheme resulting from them. In other words, I doubt many people really "Want to want X." They often convince themselves of this in order to enable fulfilling some other directive.

0Vladimir_Nesov16y

Of two conflicting desires, we call second-order the one we don't expect to go away, and as such more invariant, part of self, even if not ever in control. Most second-order desires are not about first-order desires, they are about the same thing as the first-order desire. For the second-order desire, modifying the first-order desire is instrumental, not terminal, and the same applies in the other direction. The differences I see come from first-order desire being actually in control, and being stupid enough not to work on eliminating the second-order desire.

2Psychohistorian16y

First , these two statements are 2nd order and 3rd order respectively (taboo "straight" and you get -very roughly- "Larry wants to want to have sex with women" ). Second, they are not representations of the same thing, since they point in opposite directions. Thus, they don't seem to support the claim that first and second order desires can switch places. More importantly, you can't just infer n-th order desires from n-m, n>m>0 order desires. "I want chocolate," does not imply: "I do not want to want to want to not want to want to not want to not want to want to want to not want to not want chocolate," even though the two happen to point in the same direction. The first one is true, the second one is almost certainly false, since I really don't think that hard about chocolate. Higher-order desires get really, really convoluted if you use imprecise language, and they're not exactly simple to begin with. I am starting to agree with other posters that the whole construction may not map reality too accurately, but if you actually use precise language, n-th order desires are distinct and meaningful. Without precise language, you're just showing it's possible to say the same thing in more than one way, which is true, but not insightful.

0Jordan16y

I agree. Generally people don't have very high order desires. Discounting the accuracy/usefulness of the notion of ordered desires was the entire thrust of my original comment. This is still a matter of interpretation based on wording. In my original phrasing it's more apparent that the desire is first order. My set up was that Larry "experiences genuine disgust when he thinks about homosexual acts", the intent being that his reaction to homosexuality is involuntary, and his desire is simply to avoid that unpleasant reaction. This is first order. I'm not claiming the order can always be reversed. I was just giving a particular construction where it could.

0conchis16y

Isn't that just second order? Third order would be wanting to want to want heroin.

0Psychohistorian16y

It included the next sentence (wanting to want to not want); edited to make it less ambiguous.

0conchis16y

Ah. Sorry!

0[anonymous]16y

Why's that second order?

[-]pjeby16y20

There's a simpler model for all of these examples -- you're describing conflicts between an "away-from" motivation and a "towards" motivation. These systems are semi-independent, via affective asynchrony. The second-order want is then arising as a subgoal of the currently-active goal (be alert, etc.).

I guess what I'm trying to say here is that there really aren't "second order wants" in the system itself; they're just an emergent property of a system with subgoals that explicitly models itself as an agent, especially if it a... (read more)

1JamesAndrix16y

I identified 2 and 4 as most clearly about just wanting utilons. I suspect that all such metawants can be reduced to trade-offs in the world. There is a bad tasting alertness potion, an addictive happiness drug with side effects, and a nonintuitive offer of money. 3 is a bit harder to look at this way, I think any solution needs to work just as well for Darryl who is also homosexual and finds it repulsive, but instead desires to no longer find it repulsive.

[-]MugaSofer13y00

Carbonated beverages make my mouth hurt. I have developed a more generalized aversion to them after repeatedly trying to develop a taste for them and experiencing pain every time.

Wait, that's unusual? I used to have the exact same problem, but I thought it was due to generalized willpower issues. When I got better at willpower, the problem disappeared (although I still tend to choose non-carbonated versions of drinks I like if possible.)

[-]Curiouskid14y00

I'm glad you introduced me to the term meta-wanting because it reminds me on an argument against free will.

Basically, you can go to a CD store (itunes now) and you can choose which CD you choose to buy because you prefer that CD. But you cannot prefer to prefer that CD. You simply prefer (1st order) that CD. You could try to raise the order of your preferences (an idea that had not occurred to me until now), but at the next highest order, your decision has already been made.

To me, that is the most convincing argument against free will that I've ever come across. Has anyone heard it before?

1Mati_Roy11y

I want that my highest metawanting be this sentence. 1. This is my highest order of metawanting. 2. It was determine by me wanting it (so it wasn't already made). I'm joking. I don't really want to want that my highest metawanting be wanting that my highest metawanting be wanting that my highest metawanting be wanting that.... haaaaaaaaaaaaaaaaaaaaa. :-)

0Shmi13y

Have you considered that the free will debate is vacuous, as, ironically, we have no choice but to act as if we had free will?

0Curiouskid13y

So, I made this post before I'd even read the sequences.

[-]JamesCole16y00

I don't think the right way to clarify this problem is by looking at it terms of first- and second-level desires. I think you need to turn it around and see it as a matter of what 'true self' means.

If people say that the desires you "endorse" on the second level are most reflective of your true self, they're wrong. This is because what we take to define our true selfs is based on different criteria, and those criteria define it such that people's second-level desires don't always match up with what we taken their 'true selves' to be, as in the case of Larry.

[-]Drahflow16y00

In a perfectly rational agent, no n-th order wants should exist.

Your problems with mountain dew might account for -1 util, you being awake for 2 utils, then you "want" to drink that stuff. Shut up and add.

The only source of multi-level desires I can see is an imperfect caching algorithm, which spews forth "Do not drink mountain dew" although the overall utility would be positive.

2conchis16y

It still seems perfectly reasonable for a rational agent to not-want to not-want Mountain Dew here. If it were feasible to self-modify to become Moutain-Dew-indifferent at a cost less than 1 util, then the utility of drinking Mountain Dew & being indifferent to it would be greater than the utility of drinking it while continuing to not-want it.

1steven046116y

Self-modifying not to feel discomfort from drinking Mountain Dew isn't at all the same thing as self-modifying to want to drink Mountain Dew keeping discomfort constant. The latter isn't something you want to do on pain of committing the Wirehead Fallacy. The former isn't something that "want to want" language really applies to, as far as I can see.

1Alicorn16y

I don't see why not. Spicy foods inflict pain; wanting to develop a greater tolerance for cayenne pepper wouldn't be all that weird. Mountain Dew inflicts pain; learning to put up with that would be instrumentally useful to me. I'd prefer that it just not inflict pain, but assuming that's not an option, I'd be okay with just developing the ability to deal with it.

2stcredzero16y

In case it hasn't been mentioned, many stores that sell Mountain Dew also stock NoDoz tablets, which you can develop a technique for swallowing whole with water. I was actually in the position of wanting to develop a tolerance for spicy foods when I was young. I would go to the pantry and treat myself by dosing myself with Tabasco sauce. But I don't think of that as my "wanting to want KimChee." I wanted to eat KimChee like my parents and I took steps to achieve that goal. I don't think there's any need for meta-desires at all, except for one: a drive to resolve conflicting desires. And this is arguably also a 1st order desire. There are obvious reasons why we'd evolve such a drive. We can explain away things like wanting to want internal conflict as merely a desire to be someone like Timothy Levitch from "Waking Life." It also makes sense that one can formulate paradoxical desires that are strongly resistant to resolution, or rationalize to oneself that they are conundrums. But I posit that these are merely ill-formed desires -- that if you can't reduce everything down to conflicting 1st-order desires, you haven't delved deeply enough. http://dannyman.toldme.com/2003/11/19/timothy-levitch-waking-life/

0steven046116y

Agreed, what I said wasn't literally true. If you get 2 utils from caffeine and -1 util from pain, and if despite this you don't want to drink MD, then it's rational to self-modify to want to drink MD. But the point I meant to make is that it's not rational to self-modify to assign 0 utils to the same pain instead (because you don't care about utils, you care about things using utils), which is what I (mis)interpreted conchis as saying.

0conchis16y

I interpret "wanting to want" as encompassing the former, and don't really see any reason to limit it in the way you suggest. But either way, we have no substantive disagreement.

2latanius16y

Or the vertical hierarchy in our brains, and our inability to modify it. All four examples show "first order desires" created by lower level subsystems (taste perception, sexuality, motivations, etc.), and the higher level ones are trying to override it (consciousness, planning, models of the future world). But we aren't designed rationally enough to be able to control those lower levels, although we know partially about what they "think". ("We" = the conscious part writing LW comments...)

[-]dclayh16y00

So far so good. I look forward to the hard stuff :) And thanks for engaging my request.

Actually, your calling second-level agreement "endorsement" has led me to wonder whether there's a special term for desires that you want to want, want to want to want, and so on ad infinitum, analogous to common knowledge) or Hofstadter's hyperrational groups (where everyone knows that everyone knows etc. that everyone is rational).

1Peter_de_Blanc16y

"Reflectively consistent."

0dclayh16y

I think that's for beliefs, not desires. Particularly because you can bring your beliefs and metabeliefs, etc., into alignment by reflection, whereas making your desires consistent requires at a minimum some kind of action, and may not be possible at all (except for the trivial case of instrumental second-order desires as in Ex. 1).

1Alicorn16y

I haven't run into any special jargon for endorsed desires, but it would be a cool word to have. There's some debate about whether we can really go up indefinitely in higher orders - it's not clear that we have the necessary cognitive capacity to go higher than about six nested intentional states. (For instance, I intend that you know that I intend that you know that I want to want Mountain Dew, but if you could come up with a more complicated string of propositional attitudes, I'd be a bit lost.)

0stcredzero16y

I see nothing more than the intent to share knowledge of intent, or more concisely, to share intent. This can be reduced to a desire for synchrony of consciousness. When I sit down to play music with others, I do not have all of these intents and meta-intents. Yet it's arguably true that I do intend for the other musician to know that I intend him or her to know that I want to make good music. One can compose an infinite number of such statements concerning meta-intent and knowledge. But this is only an effect of the linearity of language, causing us to consider each constituent relationship one at a time. Beyond this, one can simply deal with the whole (jam session) in a highly functional, immediate, and mutually pleasing way. Nth order desires are really just a variation on Xeno's paradox. It's a nifty mental exercise, but reality doesn't work that way. One just has 1st order wants, and a conflict resolution drive which is the only 2nd order want.

0dclayh16y

Update: upon reading Frankfurt I find that he calls this sort of infinitely regressed metawanting a "decisive" want or desire.

0cousin_it16y

Seems to be related to FAI. I'd look forward very much to any mathematic formalization of the term, if you have ideas how to get there.

[-][anonymous]16y-10

Suppose also that there is a can of Mountain Dew handy. I know that Mountain Dew contains caffeine and that caffeine will make me alert.

I am hesitant to bring it up because I don't want to become the multiculturalism police on LessWrong, but I found this distracting. American Mountain Dew contains a large caffeine content yet in most other countries Mountain Dew is Caffeine free. There is a significant minority of LessWrong participants who do not dwell in America and those readers can not help but become distracted when posts seem to be clearly inten... (read more)

2Alicorn16y

I'm sure you thought this would be cute or funny or something, but the objections aren't commensurate. I wasn't making a sweeping statement about the alertness-giving properties of Mountain Dew. Trying to do that would have been beside the point, since I was giving an individual example about my own beverage-related limitations, and living where I live, a Mountain Dew that might materialize in my home would have plenty of caffeine. To compare, I wouldn't have blinked if Psychohistorian had phrased the original remark about women as "I'll still find women alluring", making it about himself instead of about women. Alternatively, I could just protest that in my idiolect, Mountain Dew refers to a beverage that contains caffeine, and your wacky foreign Mountain Dew is not Mountain Dew at all.

-3[anonymous]16y

Where you "attribute [your] distraction entirely to the sense that it was directed at a presumed male audience", I attribute my distraction entirely to the sense that it was directed at a presumed American audience. It is regarding this presumption that I demand commensurate consideration. Such consideration could perhaps take the form of a simple acknowledgement: "Oh, really? I never new that! Next time I'll either use a different example or I'll throw in brief a parenthetised comment or footnote to make the text accessible to the non-American reader." You could make that protestation. Yet you are dismissing the alternate experience as 'wacky foreign' as a point demonstrating how gender specific descriptions is incommensurably more significant than nationality-specificity. This is distressing. It would seem to lend support to a conclusion that your objections have been less about the implicit exclusion of minority participants and more about mere political manouvering in favour of your own particular group. This significantly reduces the credibility of any objections that you may make, at least in my eyes. I could quite fairly be accused of having an anti-hypocricy bias.

3Cyan16y

Here's where your analogy runs off the rails. Alicorn's text isn't directed at a presumed American audience -- it's directed at an audience presumed to be able to infer that Mountain Dew contains caffeine where she lives. Your rejoinder skips over this exact point made by Alicorn:

0[anonymous]16y

My rejoinder did not so much skip the point as not see the point as significant. One of the strengths of analogies is that they can help trace where exactly the difference in thinking or opinion lies. I actually don't see Psycho's presumption of audience as more significant than that of Alicorn; I can infer that Psycho is speaking from his individual experience as a male just as easily as that Ali is speaking from her individual experience as an American.

2JGWeissman16y

The difference is that Phycohistorian was describing experiences that he intended the audience to recognize and identify with as their own, while Alicorn was describing her own experience as her own unique experience.

2[anonymous]16y

That does seem like a bizarre thing for Psycho to intend! I gave him a little more benefit of the doubt. Perhaps I was too generous in my interpretation, I get that a lot!

3SoullessAutomaton16y

It isn't necessarily a deliberate, conscious intent. However: vs. Surely you see the difference?

2Alicorn16y

I'm sorry if it wasn't clear, but the choice of the words "wacky foreign" was to be silly (in an apparently failed attempt to keep the discussion light), not to indicate an actual belief about the relative wackiness of foreign and domestic soft drinks.

-1[anonymous]16y

I am afraid I missed that. The plain literal interpretation actually seemed to me to more closely fit the remainder of the reply. The core of your reply took the discussion from 'light and slightly silly' to serious while simultaniously dismissing the underlying serious message, which rather shocked me. 'Light and silly' just doesn't seem to work when rapport is broken, which I tend to discover rather often! In any case, I wouldn't change a word of either of my preceeding posts yet am intrigued by the response. (And also somewhat glad karma can be gained so readily on trivial topics that it can be freely spent on those that I feel actually matter.)

0[anonymous]16y

There's quite a difference between a parochialism that is, at worst, confusing, and a parochialism that makes some people feel ignored or excluded (regardless of whether or not this feeling is justified).

1Nick_Tarleton16y

There's a big difference between parochialism that is, at worst, confusing, and parochialism that makes some people feel ignored or excluded (even if they aren't being ignored or excluded).

1[anonymous]16y

Parochialism exhibits itself perhaps most significantly when it comes to deciding which claims of exclusion are socially acceptable to make.

3SoullessAutomaton16y

You realize that this is an anti-applause light that conveys little informational value, right? The actual argument is that since women comprise half the population and are severely underrepresented on LW as it is, if phrasing that implies the audience is uniformly male makes women feel excluded it is detrimental to the goal of spreading rationality. Do you actually have an argument against this? Please note that "women won't actually feel excluded" is demonstrably false.

0[anonymous]16y

No. It adequately serves as a descriptive reference to the social dynamics involved in determining what is Right, moral, acceptable enlightened or otherwise good. If you can suggest a substitute phrase then I would happily adopt it. Arguments along the lines of 'something to do with political correct therefore something bad about the other side' are common. It is to be expected that some will assume a similar error of reasoning is being applied whenever the phrase is being used no matter the actual content and I would prefer to have a phrase that avoided this hassle. No. I can see a few minor arguments that could be made but why would I make them? It is a conclusion that I support. However, I find some of the soldiers used to support said conclusion distasteful, inconsistent in their application and neglecting some of the spirit of compromise and mutual understanding necessary when communicating across a cultural barrier. I reject posts and certain of the normative demands contained therein on their own merit as I see it. While it doesn't make any difference for the purposes of my reply, that is not the 'actual conclusion'. It is reasonable to desire an inclusive environment independently of the influence this improved environment may have on the spread of rationality. I for one accept inclusiveness a terminal value while 'spreading rationality' is not a goal of mine at all. That claim in the quotes would be an insane claim to make.

2SoullessAutomaton16y

The phrase you're looking for is probably "socially acceptable" or "social norm". The phrase "politically correct" is primarily used as a connotationally-loaded derogatory for social norms the speaker disagrees with, and to signal, in a beliefs-as-attire manner, group membership with certain political positions. If you want to criticize social norms, which I agree is a rewarding and enjoyable hobby, you would do better to name the specific norms you take issue with, rather than using a catch-all term for norms you dislike. For instance: Which claims of exclusion are not acceptable to raise that you think ought to be? Why? What norms would you prefer? Okay. Which specific normative demands are you rejecting? So you accept inclusiveness as valuable, but disagree with explanations given by individuals who felt excluded for why they felt that way, and feel that others are neglecting the spirit of mutual understanding across a cultural barrier? I'm not sure I follow.

0[anonymous]16y

Socially acceptable is suitable, I edited. In most situations I avoid it since it is an applause light that is still yet to be diffused. In this context, however, it feels more like a neutral descriptor. Do not presume so much.

1SoullessAutomaton16y

Then I confess I am at a loss as to what your point in all this was, as you seem to have stated is a rejection of something that other people said without any real explanation as to what you're rejecting, or why.

1[anonymous]16y

As I stated, I intend no point other than those particular assertions made in my posts. If you insist that I must only deploy arguments in support of a particular political agenda then said agenda is this: Bad arguments and hypocrisy presented in support of positions I approve of are still bad arguments and hypocrisy.

6SoullessAutomaton16y

I don't normally remark on such things, but I'm a bit discouraged to note the following: * The parent comment was the first time Cameron_Taylor has posted anything in roughly a month, in a long-dead argument in which he and I were disagreeing. * At roughly the same time the parent comment was posted, roughly the last 80 or so posts I've made were all voted down, consecutively, once each, for no discernable reason. * I recall at least two other commenters mentioning being voted down suddenly on multiple unrelated comments previously while arguing with Cameron_Taylor. Karma is easy-come, easy-go, but I'm thinking that someone is not exactly participating in good faith here.

5conchis16y

By way of confirmation, this has indeed happened to both myself and at least one other commenter previously (I'll leave it to them whether they want to reveal themselves). I had been waiting to see whether it would happen again to be sure, but we now seem to have pretty good evidence of bad faith. As SoullessAutomaton notes, the karma itself is not much of an issue, but it's nonetheless rather disappointing to see this sort of behavior. It's not immediately clear whether there's much to be done about it other than public shaming, but as a possible means of preventing this happening again, I don't suppose there's any way to revoke the downvoting privileges of those who seem to be abusing the system?

3Alicorn16y

I was the other commenter, and confirm the observation.

0Mike Bishop16y

This is unfortunate, perhaps there should be a top level post to discuss the wise way to respond.

1thomblake16y

For the record, I've been known to downvote large numbers of posts at once (since I'm only here looking at comments for short periods of time, and downvote a lot of posts) but I read them first. Not so much lately, due to the extremely limited number of downvotes available.

4SoullessAutomaton16y

While I do not profess to understand the motivation for it, your apparent conviction that a substantial percentage of comments ought to be voted down is of an entirely different character than a mass downvoting aimed at a specific person, targetting what seems to be all of their comments from the past three weeks or so. The latter kind of behavior I would expect on sites like Digg; I tend to expect better of people here.

3Mike Bishop16y

Let's consider a less convenient possible world. I come across several stupid comments, realize that the author has a lot of karma, and then start reading their old comments. Careful reading, including the context when necessary, leads me to believe half of their old comments are bad or overrated, and deserving of a downvote. I would argue that making those downvotes is justified, but I'd like to think I have better things to do than read and vote on comments on dead threads. Edit: this comment may be confusing, please read my follow up to orthonormal.

1orthonormal16y

Don't use "least convenient possible world" to mean "a different hypothesis to explain what you're seeing". We don't want the usage to get confused. EDIT: Also, it's unlikely for this effect to result in every one of SA's last 80 comments being downvoted once.

0Mike Bishop16y

I'm sorry I was unclear. I didn't mean to suggest that this was an alternative explanation for this event. In fact, as you point out, the hypothetical I described contradicts SA's testimony in an important way (the proportion of the comments downvoted). The reason I brought up the hypothetical was to promote discussion about scenarios that are more difficult to evaluate than what actually appears to have occurred.

0SoullessAutomaton16y

The reason the subject came up at all is because this instance was particularly blatant. Otherwise, we don't generally have enough information to evaluate other scenarios reliably--this is why Eliezer wants a way to monitor voting abuse. Even so I'm willing to grant that it could be something innocuous (and will apologize if that is the case), but the evidence so far leans toward abuse. If you want to promote discussion about the issue, a top-level post is probably in order, as you yourself previously noted; feel free to make one.

0Eliezer Yudkowsky16y

I've previously asked Tricycle for the ability to monitor this sort of thing. I will ask them again. For the record, this sort of systematic downvoting is not only not in good faith, but grounds for removal of the ability to downvote.

0[anonymous]16y

Those that I specifically reject in any specific post that I make. I have neither the obligation nor inclination to use my posts to make a united stance for one particular political position that I identify with. I in fact choose to disagree with poor arguments for opinions I approve of. I don't believe following is your intent.

0SoullessAutomaton16y

This actually loses something in context, though--Mt. Dew (in the USA) has a somewhat higher caffeine content than those (about 20% more, I think), and also has a reputation as something people drink primarily for the caffeine content, not the flavor. A better example might be "energy drinks" like Red Bull, which are typically dense, syrupy carbonated beverages with twice the caffeine content of a cola, but I'm not sure how common those are in other areas. EDIT: This comment was written on the premise that the parent was a genuine request for greater acknowledgement of LW readers not in the USA, not a disingenuous attempt to make an off-topic point about something completely unrelated to this post. Disregard this comment as appropriate.

0[anonymous]16y

In Australia at least we have Red Bull and that does seem to be a better substitute. The grandparent was a genuine request. The sincerity of the disingenuous EDIT in the reply I have ironic doubts about.

1SoullessAutomaton16y

I reevaluated the comment after seeing the downvotes it received and based on the apparent attempt to score a point in a discussion on a different post. If you were being genuine then okay, I accept your word on the matter and retract the edit with my apologies.

Moderation Log