Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Pluralistic Moral Reductionism

34 Post author: lukeprog 01 June 2011 12:59AM

Part of the sequence: No-Nonsense Metaethics

Disputes over the definition of morality... are disputes over words which raise no really significant issues. [Of course,] lack of clarity about the meaning of words is an important source of error… My complaint is that what should be regarded as something to be got out of the way in the introduction to a work of moral philosophy has become the subject matter of almost the whole of moral philosophy...

Peter Singer

 

If a tree falls in the forest, and no one hears it, does it make a sound? If by 'sound' you mean 'acoustic vibrations in the air', the answer is 'Yes.' But if by 'sound' you mean an auditory experience in the brain, the answer is 'No.'

We might call this straightforward solution pluralistic sound reductionism. If people use the word 'sound' to mean different things, and people have different intuitions about the meaning of the word 'sound', then we needn't endlessly debate which definition is 'correct'.1 We can be pluralists about the meanings of 'sound'. 

To facilitate communication, we can taboo and reduce: we can replace the symbol with the substance and talk about facts and anticipations, not definitions. We can avoid using the word 'sound' and instead talk about 'acoustic vibrations' or 'auditory brain experiences.'

Still, some definitions can be wrong:

Alex: If a tree falls in the forest, and no one hears it, does it make a sound?

Austere MetaAcousticist: Tell me what you mean by 'sound', and I will tell you the answer.

Alex: By 'sound' I mean 'acoustic messenger fairies flying through the ether'.

Austere MetaAcousticist: There's no such thing. Now, if you had asked me about this other definition of 'sound'...

There are other ways for words to be wrong, too. But once we admit to multiple potentially useful reductions of 'sound', it is not hard to see how we could admit to multiple useful reductions of moral terms.

 

Many Moral Reductionisms

Moral terms are used in a greater variety of ways than sound terms are. There is little hope of arriving at the One True Theory of Morality by analyzing common usage or by triangulating from the platitudes of folk moral discourse. But we can use stipulation, and we can taboo and reduce. We can use pluralistic moral reductionism2 (for austere metaethics, not for empathic metaethics).

Example #1:

Neuroscientist Sam Harris: Which is better? Religious totalitarianism or the Northern European welfare state?

Austere Metaethicist: What do you mean by 'better'?

Harris: By 'better' I mean 'that which tends to maximize the well-being of conscious creatures'.

Austere Metaethicist: Assuming we have similar reductions of 'well-being' and 'conscious creatures' in mind, the evidence I know of suggests that the Northern European welfare state is more likely to maximize the well-being of conscious creatures than religious totalitarianism.

Example #2:

Philosopher Peter Railton: Is capitalism the best economic system?

Austere Metaethicist: What do you mean by 'best'?

Railton: By 'best' I mean 'would be approved of by an ideally instrumentally rational and fully informed agent considering the question ‘How best to maximize the amount of non-moral goodness?' from a social point of view in which the interests of all potentially affected individuals are counted equally.

Austere Metaethicist: Assuming we agree on the meaning of 'ideally instrumentally rational' and 'fully informed' and 'agent' and 'non-moral goodness' and a few other things, the evidence I know of suggests that capitalism would not be approved of by an ideally instrumentally rational and fully informed agent considering the question ‘How best to maximize the amount of non-moral goodness?' from a social point of view in which the interests of all potentially affected individuals were counted equally.

Example #3:

Theologian Bill Craig: Ought we to give 50% of our income to efficient charities?

Austere Metaethicist: What do you mean by 'ought'?

Craig: By 'ought' I mean 'approved of by an essentially just and loving God'.

Austere Metaethicist: Your definition doesn't connect to reality. It's like talking about atom-for-atom 'indexical identity' even though the world is made of configurations and amplitudes instead of Newtonian billiard balls. Gods don't exist.

But before we get to empathic metaethics, let's examine the standard problems of metaethics using the framework of pluralistic moral reductionism.

 

Cognitivism vs. Noncognitivism

One standard debate in metaethics is cognitivism vs. noncognitivism. Alexander Miller explains:

Consider a particular moral judgement, such as the judgement that murder is wrong. What sort of psychological state does this express? Some philosophers, called cognitivists, think that a moral judgement such as this expresses a belief. Beliefs can be true or false: they are truth-apt, or apt to be assessed in terms of truth and falsity. So cognitivists think that moral judgements are capable of being true or false. On the other hand, non-cognitivists think that moral judgements express non-cognitive states such as emotions or desires. Desires and emotions are not truth-apt. So moral judgements are not capable of being true or false.3

But why should we expect all people to use moral judgments like "Stealing is wrong" to express the same thing?4

Some people who say "Stealing is wrong" are really just trying to express emotions: "Stealing? Yuck!" Others use moral judgments like "Stealing is wrong" to express commands: "Don't steal!" Still others use moral judgments like "Stealing is wrong" to assert factual claims, such as "stealing is against the will of God" or "stealing is a practice that usually adds pain rather than pleasure to the world."

It may be interesting to study all such uses of moral discourse, but this post focuses on addressing cognitivists, who use moral judgments to assert factual claims. We ask: Are those claims true or false? What are their implications?

 

Objective vs. Subjective Morality

Is morality objective or subjective? It depends which moral reductionism you have in mind, and what you mean by 'objective' and 'subjective'.

Here are some common5 uses of the objective/subjective distinction in ethics:

  • Moral facts are objective1 if they are made true or false by mind-independent facts, otherwise they are subjective1.
  • Moral facts are objective2 if they are made true or false by facts independent of the opinions of sentient beings, otherwise they are subjective2.
  • Moral facts are objective3 if they are made true or false by facts independent of the opinions of humans, otherwise they are subjective3.

Now, consider Harris' reduction of morality to facts about the well-being of conscious creatures. His theory of morality is objectiveand objective2, because facts about well-being are independent of anyone's opinion. Even if the Nazis had won WWII and brainwashed everybody to have the opinion that torturing Jews was moral, it would remain true that torturing Jews does not increase the average well-being of conscious creatures. But Harris' theory of morality is not objective1, because facts about the well-being of conscious creatures are mind-dependent facts.

Or, consider Craig's theory of morality in terms of divine approval. His theory doesn't connect to reality, but still: is it objective or subjective? Craig's theory says that moral facts are objective3, because they don't depend on human opinion (God isn't human). But his theory doesn't say that morality is objective2 or objective1, because for him, moral facts depend on the opinion of a sentient being: God.

A warning: ambiguous terms like 'objective' and 'subjective' are attractors for sneaking in connotations. Craig himself provides an example. In his writings and public appearances, Craig insists that only God-based morality can be objective.6 What does he mean by 'objective'? On a single page,7 he uses 'objective' to mean "independent of people's opinions" (objective2) and also to mean "independent of human opinion" (objective3). I'll assume he means that only God-based morality can be objective3, because God-based morality is clearly not objective2 (Craig's God is a person, a sentient being).

And yet, Craig says that we need God in order to have objective3 morality as if this should be a big deal. But hold on. Even a moral code defined in terms of the preferences of Washoe the chimpanzee is objective3. So not only is Bill's claim that only God-based morality can be objectivefalse (because Harris' moral theory is also objective3), but also it's trivially easy to come up with a moral theory that is 'objective' in Craig's (apparent) sense of the term (that is, objective3).8

Moreover, Harris' theory of morality is objective in a 'stronger' sense than Craig's theory of morality is. Harris' theory is objective3 and objective2, while Craig's theory is merely objective3. Whether he's doing it consciously or not, I wonder if Craig is using the word 'objective' to try to sneak in connotations that don't actually apply to his claims once you pay attention to what Craig actually means by the word 'objective'. If Craig told his audience that we need God for morality to be 'objective' in the same sense that morality defined in terms of the preferences of a chimpanzee is 'objective', would this still still have his desired effect on his audience? I doubt it.

Once you've stipulated your use of 'objective' and 'subjective', it is often trivial to determine whether a given moral reductionism is 'objective' or 'subjective'. But what of it? What force should those words carry after you've tabooed them? Be careful not to sneak in connotations that don't belong.

 

Relative vs. Absolute Morality

Is morality relative or absolute? Again, it depends which moral reductionism you have in mind, and what you mean by 'relative' and 'absolute'. Again, we must be careful about sneaking in connotations.

 

Moore's Open Question Argument

"He's an unmarried man, but is he a bachelor?" This is a 'closed' question. The answer is obviously "Yes."

In contrast, said G.E. Moore, all questions of the type "Such and such is X, but is it good?" are open questions. It feels like you can always ask, "Yes, but is it good?" In this way, Moore resists the identification of 'morally good' with any set of natural facts. This is Moore's Open Question Argument. Because some moral reductionisms do identify 'good' or 'right' with a particular X, those reductionisms had better have an answer to Moore.

The Yudkowskian response is to point out that when cognitivists use the term 'good', their intuitive notion of 'good' is captured by a massive logical function that can't be expressed in simple statements like "maximize pleasure" or "act only in accordance with maxims you could wish to be a universal law without contradiction." Even if you think everything you want (or rather, want to want) can be realized by (say) maximizing the well-being of conscious creatures, you're wrong. Your values are more complex than that, and we can't see the structure of our values. That is why it feels like an open question remains no matter which simplistic identification of "Good = X" you choose.

The problem is not that there is no way to identify 'good' or 'right' (as used intuitively, without tabooing) with a certain X. The problem is that X is huge and complicated and we don't (yet) have access to its structure.

But that's the response to Moore after righting a wrong question - that is, when doing empathic metaethics. When doing mere pluralistic moral reductionism, Moore's argument doesn't apply. If we taboo and reduce, then the question of "...but is it good?" is out of place. The reply is: "Yes it is, because I just told you that's what I mean to communicate when I use the word-tool 'good' for this discussion. I'm not here to debate definitions; I'm here to get something done."9

 

The Is-Ought Gap

(This section rewritten for clarity.)

Many claim that you cannot infer an 'ought' statement from a series of 'is' statements. The objection comes from Hume, who said he was surprised whenever an argument made of is and is not propositions suddenly shifted to an ought or ought not claim, without explanation.10

The solution is to make explicit the bridge from 'ought' statements to 'is' statements.

Perhaps the arguer means something non-natural by 'ought', such as 'commanded by God' or 'in accord with irreducible, non-natural facts about goodness' (see Moore). If so, I would reject that premise of the argument, because I'm a reductionist. At this point, our discussion might need to shift to a debate over the merits of reductionism.

Or perhaps by 'you ought to X' the arguer means something fully natural, such as:

  • "X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z."
  • Or: "X tends to maximizes reward signals in agents exhibiting multiple-drafts consciousness" (or, as Sam Harris more broadly puts it, "X tends to maximize well-being in conscious creatures").
  • Or: "X is what a Bayes-rational and Hubble-volume-omniscient agent would do if it was motivated to maximize the amount of non-moral goodness from a view in which the interests of all potentially affected individuals were counted equally, where 'non-moral goodness' refers to what an agent would want if it were he to contemplate its present situation from a standpoint fully and vividly informed about itself and its circumstances, and entirely free of cognitive error or lapses of instrumental rationality" (see Railton's metaethics).
  • Or: "X maximizes the complicated function that can be computed by extrapolating (in a particular way) the motivations encoded by my brain" (see CEV).
  • Or: "[insert here whatever statement, if believed, would motivate one to do X]" (see Will Sawin).

Or, the speaker may have in mind a common ought-reductionism known as the hypothetical imperative. This is an ought of the kind: "If you desire to lose weight, then you ought to consume fewer calories than your burn." (But usually, people leave off the implied if statement, and simply say "You should eat less and exercise more.")

A hypothetical imperative (as some use it) can be translated from 'ought' to 'is' in a straightforward way: "If you desire to lose weight, then you ought to consume fewer calories than you burn" translates to the claim "If you consume fewer calories than you burn, then you will (or are, ceteris paribus, more likely to) fulfill your desire to lose weight."11

Or, the speaker may be using 'ought' to communicate something only about other symbols (example: Bayes' Rule), leaving the bridge from 'ought' to 'is' to be built when the logical function represented by his use of 'ought' is plugged into a theory that refers to the world.

But one must not fall into the trap of thinking that a definition you've stipulated (aloud or in your head) for 'ought' must match up to your intended meaning of 'ought' (to which you don't have introspective access). In fact, I suspect it never does, which is why the conceptual analysis of 'ought' language can go in circles for centuries, and why any stipulated meaning of 'ought' is a fake utility function. To see clearly to our intuitive concept of ought, we'll have to try empathic metaethics (see below).

But whatever our intended meaning of 'ought' is, the same reasoning applies. Either our intended meaning of 'ought' refers (eventually) to the world of math and physics (in which case the is-ought gap is bridged), or else it doesn't (in which case it fails to refer).12

 

Moral realism vs. Anti-realism

So, does all this mean that we can embrace moral realism, or does it doom us to moral anti-realism? Again, it depends on what you mean by 'realism' and 'anti-realism'.

In a sense, pluralistic moral reductionism can be considered a robust form of moral 'realism', in the same way that pluralistic sound reductionism is a robust form of sound realism. "Yes, there really is sound, and we can locate it in reality — either as vibrations in the air or as mental auditory experiences, however you are using the term." In the same way: "Yes, there really is morality, and we can locate it in reality — either as a set of facts about the well-being of conscious creatures, or as a set of facts about what an ideally rational and perfectly informed agent would prefer, or as some other set of natural facts."

But in another sense, pluralistic moral reductionism is 'anti-realist'. It suggests that there is no One True Theory of Morality. (We use moral terms in a variety of ways, and some of those ways refer to different sets of natural facts.) And as a reductionist approach to morality, it might also leave no room for moral theories which say there are universally binding moral rules for which the universe (e.g. via a God) will hold us accountable.

What matters are the facts, not whether labels like 'realism' or 'anti-realism' apply to 'morality'.

 

Toward Empathic Metaethics

But pluralistic moral reductionism satisfies only a would-be austere metaethicist, not an empathic metaethicist.

Recall that when Alex asks how she can do what is right, the Austere Metaethicist replies:

Tell me what you mean by 'right', and I will tell you what is the right thing to do. If by 'right' you mean X, then Y is the right thing to do. If by 'right' you mean P, then Z is the right thing to do. But if you can't tell me what you mean by 'right', then you have failed to ask a coherent question, and no one can answer an incoherent question.

Alex may reply to the Austere Metaethicist:

Okay, I'm not sure exactly what I mean by 'right'. So how do I do what is right if I'm not sure what I mean by 'right'?

The Austere Metaethicist refuses to answer this question. The Empathic Metaethicist, however, is willing to go the extra mile. He says to Alex:

You may not know what you mean by 'right.' But let's not stop there. Here, let me come alongside you and help decode the cognitive algorithms that generated your question in the first place, and then we'll be able to answer your question. Then we can tell you what the right thing to do is.

This may seem like too much work. Would we be motivated to decode the cognitive algorithms producing Albert and Barry's use of the word 'sound'? Would we try to solve 'empathic meta-acoustics'? Probably not. We can simply taboo and reduce 'sound' and then get some work done.

But moral terms and value terms are about what we want. And unfortunately, we often don't know what we want. As such, we're unlikely to get what we really want if the world is re-engineered in accordance with our current best guess as to what we want. That's why we need to decode the cognitive algorithms that generate our questions about value and morality.

So how can the Empathic Metaethicist answer Alex's question? We don't know the details yet. For example, we don't have a completed cognitive neuroscience. But we have some ideas, and we know of some open problems that may admit of progress once more people understand them. In the next few posts, we'll take our first look at empathic metaethics.13

 

Previous post: Conceptual Analysis and Moral Theory

 

 

Notes

1 Some have objected that the conceptual analysis argued against in Conceptual Analysis and Moral Theory is not just a battle over definitions. But a definition is "the formal statement of the meaning or significance of a word, phrase, etc.", and a conceptual analysis is (usually) a "formal statement of the meaning or significance of a word, phrase, etc." in terms of necessary and sufficient conditions. The goal of a conceptual analysis is to arrive at a definition for a term that captures our intuitions about its meaning. The process is to bash our intuitions against others' intuitions until we converge upon a set of necessary and sufficient conditions that captures them all. But consider Barry and Albert's debate over the definition of 'sound'. Why think Albert and Barry have the same concept in mind? Words mean slightly different things in different cultures, subcultures, and small communities. We develop different intuitions about their meaning based on divergent life experiences. Our intuitions differ from each other's due to the specifics of unconscious associative learning and attribution substitution heuristics. What is the point of bashing our intuitions about the meaning of terms against each other for thousands of pages, in the hopes that we'll converge on a precise set of necessary and sufficient conditions? Even if we can get Albert and Barry to agree, what happens when Susan wants to use the same term, but has slightly differing intuitions about its meaning? And, let's say we arrive at a messy set of 6 necessary and sufficient conditions for the intuitive meaning of the term. Is that going to be as useful for communication as one we consciously chose because it carved-up thingspace well? I doubt it. The IAU's definition of 'planet' is more useful than the folk-intuitions definition of 'planet'. Folk intuitions about 'planet' evolved over thousands of years and different people have different intuitions which may not always converge. In 2006, the IAU used modern astronomical knowledge to carve up thingspace in a more useful and informed way than our intuitions do.

A passage from Bertrand Russell (1953) is appropriate. Russell said that many philosophers reminded him of

the shopkeeper of whom I once asked the shortest way to Winchester. He called to a man in the back premises:

"Gentleman wants to know the shortest way to Winchester."

"Winchester?" an unseen voice replied.

"Aye."

"Way to Winchester?"

"Aye."

"Shortest way?"

"Aye."

"Dunno."

He wanted to get the nature of the question clear, but took no interest in answering it. This is exactly what modern philosophy does for the earnest seeker after truth. Is it surprising that young people turn to other studies?

2 Compare also to the biologist's 'species concept pluralism' and the philosopher's 'art concept pluralism.' See Uidhir & Magnus (2011). Also see 'causal pluralism' (Godfrey-Smith, 2009; Cartwright, 2007), 'theory concept pluralism' (Magnus, 2009) and, especially, 'metaethical contextualism' (Bjornsson & Finlay, 2010) or 'metaethical pluralism' or 'metaethical ambivalence' (Joyce, 2011). Joyce quotes Lewis (1989), who wrote that some concepts of value refer to things that really exist, and some concepts don't, and what you make of this situation is largely a matter of temperament:

What to make of the situation is mainly a matter of temperament. You can bang the drum about how philosophy has uncovered a terrible secret: there are no values! ... Or you can think it better for public safety to keep quiet and hope people will go on as before. Or you can declare that there are no values, but that nevertheless it is legitimate—and not just expedient—for us to carry on with value-talk, since we can make it all go smoothly if we just give the name of value to claimants that don't quite deserve it... Or you can think it an empty question whether there are values: say what you please, speak strictly or loosely. When it comes to deserving a name, there's better and worse but who's to say how good is good enough? Or you can think it clear that the imperfect deservers of the name are good enough, but only just, and say that although there are values we are still terribly wrong about them. Or you can calmly say that value (like simultaneity) is not quite as some of us sometimes thought. Myself, I prefer the calm and conservative responses. But as far as the analysis of value goes, they're all much of a muchness.

Joyce concludes that, for example, the moral naturalist and the moral error theorist may agree with each other (when adopting each other's own language):

[Metaethical ambivalence] begins with a kind of metametaethical enlightenment. The moral naturalist espouses moral naturalism, but this espousal reflects a mature decision, by which I mean that the moral naturalist doesn't claim to have latched on to an incontrovertiblerealm of moral facts of which the skeptic is foolishly ignorant, but rather acknowledges that this moral naturalism has been achieved only via a non-mandatory piece of conceptual precisification. Likewise, the moral skeptic champions moral skepticism, but this too is a sophisticated verdict: not the simple declaration that there are no moral values and that the naturalist is gullibly uncritical, but rather a decision that recognizes that this skepticism has been earned only by making certain non-obligatory but permissible conceptual clarifications.

...The enlightened moral naturalist doesn't merely (grudgingly) admit that the skeptic is warranted in his or her views, but is able to adopt the skeptical position in order to gain the insights that come from recognizing that we live in a world without values. And the enlightened moral skeptic goes beyond (grudgingly) conceding that moral naturalism is reasonable, but is capable of assuming that perspective in order to gain whatever benefits come from enjoying epistemic access to a realm of moral facts.

3 Miller (2003), p. 3.

4 I changed the example moral judgment from "murder is wrong" to "stealing is wrong" because the former invites confusion. 'Murder' often means wrongful killing.

5 Also see Jacobs (2002), starting on p. 2.

6 The first premise of one of his favorite arguments for God's existence is "If God does not exist, objective moral values and duties do not exist."

7 Craig (2010), p. 11.

8 It's also possible that Craig intended a different sense of objective than the ones explicitly given in his article. Perhaps he meant objective4: "morality is objective4 if it is not grounded in the opinion of non-divine persons."

9 Also see Moral Reductionism and Moore's Open Question Argument.

10 Hume (1739), p. 469. The famous paragraph is:

In every system of morality, which I have hitherto met with, I have always remarked, that the author proceeds for some time in the ordinary ways of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when all of a sudden I am surprised to find, that instead of the usual copulations of propositions, is, and is not, I meet with no proposition that is not connected with an ought, or an ought not. This change is imperceptible; but is however, of the last consequence. For as this ought, or ought not, expresses some new relation or affirmation, 'tis necessary that it should be observed and explained; and at the same time that a reason should be given; for what seems altogether inconceivable, how this new relation can be a deduction from others, which are entirely different from it.

11 For more on reducing certain kinds of normative statements, see Finlay (2010).

12 Assuming reductionism is true. If reductionism is false, then of course there are problems for pluralistic moral reductionism as a theory of austere (but not empathic) metaethics. The clarifications in the last three paragraphs of this section are due to discussions with Wei Dai and Vladimir Nesov.

13 My thanks to Steve Rayhawk and Will Newsome for their feedback on early drafts of this post.

 

References

Bjornsson & Finlay (2010). Metaethical contextualism defended. Ethics, 121: 7-36.

Craig (2010). Five Arguments for God. The Gospel Coalition.

Cartwright (2007).Hunting Causes and Using Them: Approaches in Philosophy and Economics. Cambridge University Press.

Godfrey-Smith (2009). Causal pluralism. In Beebee, Hitchcock, & Menzies (eds.), The Oxford Handbook of Causation (pp. 326-337). Oxford University Press.

Hume (1739). A Treatise on Human Nature. John Noon.

Finlay (2010). Normativity, Necessity and Tense: A Recipe for Homebaked Normativity. In Shafer-Landau (ed.), Oxford Studies in Metaethics 5 (pp. 57-85). Oxford University Press.

Jacobs (2002). Dimensions of Moral Theory. Wiley-Blackwell.

Joyce (2011).Metaethical pluralism: How both moral naturalism and moral skepticism may be permissible positions. In Nuccetelli & Seay (eds.), Ethical Naturalism: Current Debates. Cambridge University Press.

Lewis (1989). Dispositional theories of value. Part II. Proceedings of the Aristotelian Society, supplementary vol. 63: 113-137.

Magnus (2009). What species can teach us about theory

Miller (2003). An Introduction to Contemporary Metaethics. Polity.

Russell (1953). The cult of common usage. British Journal for the Philosophy of Science, 12: 305-306.

Uidhir & Magnus (2011). Art concept pluralism. Metaphilosophy, 42: 83-97.

Comments (313)

Comment author: asr 02 June 2011 04:39:53AM *  8 points [-]

I feel like your austere meta-ethicist is mostly missing the point. It's utterly routine for different people to have conflicting beliefs about whether a given act is moral*. And often they can have a useful discussion, at the end of which one or both participants change their beliefs. These conversations can happen without the participants changing their definitions of words like 'moral', and often without them having a clear definition at all.

[This is my first LW comment -- if I do something wrong, please bear with me]

This suggests that precise definitions or agreement about definitions isn't all that critical. But it's sometimes useful to be able to reason from stipulated and mutually agreed definitions, in which case meta-ethical speculation and reasoning is doing useful work if it offers a menu of crisp, useful, definitions that can be used in discussion of specific moral claims. Relatedly, it's also doing useful work by offering a set of definitions that help people conceptualize and articulate their personal feelings about morality, even absent a concrete first-order question.

And part of what goes into picking definitions is to understand their consequences. A philosopher is doing useful work for me if he shows me that a tempting-sounding definition of 'morality' doesn't pick out the set of things I want it to pick out, or that some other definition turns out not to refer to any clear set at all.

Many mathematical entities have multiple logically equivalent definitions, that are of different utility in different contexts. (E.g., sometimes I want to think about a circle as a locus of points, and sometimes as the solution set to an equation.) In the real world, something similar happens.

When I discuss, say, abortion, with somebody, probably there are multiple working definitions of 'moral' that could be mutually agreed upon for the purpose of the conversation, and the underlying dispute would still be nontrivial and intelligible. But some definitions might be more directly applicable to the discussion -- and philosophical reasoning might be helpful in figuring out what the consequences of various definitions are. For instance, a non-cognitive strikes me intuitively as less likely to be useful -- but I'd be open to an argument showing how it could be useful in a debate.

Probably a great deal of academic writing on meta-ethics is low value. But that's true of most writing on most topics and doesn't show that the topic is pointless. (With academics being major offenders, but not the only offenders.)

*I'm thinking of the individual personal changes in belief that went along with increased opposition to official racism in America over the course of the 20th century. Or opposition to slavery in the 19th.

Comment author: Peterdjones 02 June 2011 01:00:42PM 3 points [-]

A philosopher is doing useful work for me if he shows me that a tempting-sounding definition of 'morality' doesn't pick out the set of things I want it to pick out, or that some other definition turns out not to refer to any clear set at all.

That is an important point. People often run on examples as much as or more than they do on definitions,and if their intuitions about examples are strong, that can be used to fix their definitions (ie give them revised definitions that serve their intuitions better).

The rest of the post contained good material that needed saying.

Comment author: lukeprog 04 June 2011 11:56:29PM 1 point [-]

Welcome to Less Wrong!

Is there a part of your comment that you suspect I disagree with? Or, is there a sentence in my post with which you disagree?

Comment author: asr 05 June 2011 08:33:30AM 1 point [-]

Having had time to mull over -- I think here's something about your post that bothers me. I don't think it's possible to pinpoint a single sentence, but here are two things that don't quite satisfy me.

1) Neither your austere or empathetic meta-ethicists seem to be telling me anything I wanted to hear. What I want is a "linguistic meta-ethicist", who will tell me what other competent speakers of English mean when they use "moral" and suchlike terms. I understand that different people mean different things, and I'm fine with an answer which comes in several parts, and with notes about which speakers are primarily using which of those possible definitions.

What I don't want is a brain scan from each person I talk to -- I want an explanation that's short and accessible enough to be useful in conversations. Conventional ethics and meta-ethics has given a bunch of useful definitions. Saying "well, it depends" seems unnecessarily cautious; saying "let's decode your brain" seems excessive for practical purposes.

2) Most of the conversations I'm in that involve terms like "moral" would be only slightly advanced by having explicit definitions -- and often the straightforward terms to use instead of "moral" are very nearly as contentious or nebulous. In your own examples, you have your participants talk about "well-being" and "non-moral goodness." I don't think that's a significant step forward. That's just hiding morality inside the notion of "a good life" -- which is a sensible thing to say, but people have been saying it since Plato, and it's an approach that has problems of its own.

By the way, I do understand that I may not have been your target audience, and that the whole series of posts has been carefully phrased and well organized, and I appreciate that.

Comment author: lukeprog 05 June 2011 05:54:28PM 2 points [-]

asr,

Your linguistic metaethicist sounds like the standard philosopher doing conceptual analysis. Did you see my post on 'Conceptual Analysis and Moral Theory'?

I think conversations using moral terms would be greatly advanced by first defining the terms of the debate, as Aristotle suggested. Also, the reason 'well-being' or 'non-moral goodness' are not unpacked is because I was giving brief examples. You'll notice the austere metaethicist said things like "assuming we have the same reduction of well-being in mind..." I just don't have the space to offer such reductions in what is already a long post.

Comment author: asr 05 June 2011 06:48:53PM 2 points [-]

I would find it helpful -- and I think several of the other posters here would as well -- to see one reduction on some nontrivial question carried far enough for us to see that the process can be made to work. If I understand right, your approach requires that speakers, or at least many speakers much of the time, can reduce from disputed, loaded, moral terms to reasonably well-defined and fact-based terminology. That's the point I'd most like to see you spend your space budget on in future posts.

Definitions are good. Precise definitions are usually better than loose definitions. But I suspect that in this context, loose definitions are basically good enough and that there isn't a lot of value to be extracted by increased precision there. I would like evidence that improving our definitions is a fruitful place to spend effort.

I did read your post on conceptual analysis. I just re-read it. And I'm not convinced that the practice of conceptual analysis is any more broken than most of what people get paid to do in the humanities and social sciences . My sense is that the standard textbook definitions are basically fine, and that the ongoing work in the field is mostly just people trying to get tenure and show off their cleverness.

I don't see that there's anything terribly wrong with the practice of conceptual analysis -- so long as we don't mistake an approximate and tentative linguistic exercise for access to any sort of deep truth.

Comment author: lukeprog 08 June 2011 06:22:57AM 1 point [-]

I don't think many speakers actually have an explicit ought-reduction in mind when they make ought claims. Perhaps most speakers actually have little idea what they mean when they use ought terms. For these people, emotivism may roughly describe speech acts involving oughts.

Rather, I'm imagining a scenario where person A asks what they ought to do, and person B has to clarify the meaning of A's question before B can give an answer. At this point, A is probably forced to clarify the meaning of their ought terms more thoroughly than they have previously done. But if they can't do so, then they haven't asked a meaningful question, and B can't answer the question as given.

I would like evidence that improving our definitions is a fruitful place to spend effort.

Why? What I've been saying the whole time is that improving our definitions isn't worth as much effort as philosophers are expending on it.

I'm not convinced that the practice of conceptual analysis is any more broken than most of what people get paid to do in the humanities and social sciences.

On this, we agree. That's why conceptual analysis isn't very valuable, along with "most of what people get paid to do in the humanities and social sciences." (Well, depending on where you draw the boundary around the term 'social sciences.')

I don't see that there's anything terribly wrong with the practice of conceptual analysis...

Do you see something wrong with the way Barry and Albert were arguing about the meaning of 'sound' in Conceptual Analysis and Moral Theory? I'm especially thinking of the part about microphones and aliens.

Comment author: asr 10 June 2011 07:09:14AM 3 points [-]

I agree that emotivism is an accurate description, much of the time, for what people mean when they make value judgments. I would also agree that most people don't have a specific or precise definition in mind. But emotivism isn't the only description and for practical purposes it's often not the most useful. Among other things, we have to specify which emotion we are talking about. Not all disgust is moral disgust.

Value judgments show up routinely in law and in daily life. It would be an enormous, difficult, and probably low-value task to rewrite our legal code to avoid terms like "good cause", "unjust enrichment", "unconscionable contract", and the like. Given that we're stuck with moral language, it's a useful project to pull out some definitions to help focus discourse slightly. But we aren't going to be able to eliminate them. "Morality" and its cousins are too expensive to taboo.

We want law and social standards to be somewhat loosely defined, to avoid unscrupulous actors trying to worm their way through loopholes. We don't want to be overly precise and narrow in our definitions -- we want to leverage the judgement of judges and juries. But conversely, we do want to give them guidance about what we mean by those words. And precedent supplies one sort of guidance, and some definitions give them an additional sort of guidance.

I suspect it would be quite hard to pick out precisely what we as a society mean when we use those terms in the legal code -- and very hard to reduce them to any sort of concrete physical description that would still be human-intelligible. I would be interested to see a counterexample if you can supply one easily.

I have the sense that trying to talk about human judgement and society without moral language would be about like trying to discuss computer science purely in terms of the hardware -- possible, but unnecessarily cumbersome.

One of the common pathologies of the academy is that somebody comes up with a bright idea or a powerful intellectual tool. Researchers then spend several years applying that tool to increasingly diverse contexts, often where the marginal return from the tool is near-zero. Just because conceptual analysis is being over-used doesn't mean that it is always useless! The first few uses of it may indeed have been fairly high-value in aiding us in communicating. The fact that the tool is then overused isn't a reason to ignore it.

Endless wrangles about definitions, I think are necessarily low-value. Working out a few useful definitions or explanations for a common term can be valuable, though -- particularly if we are going to apply those terms in a quasi-formal setting, like law.

Comment author: eli_sennesh 18 November 2013 01:35:36PM 1 point [-]

I would think that the Hypothetical Imperatives are useful there. You can thus break down your own opinions into material of the form:

"If the set X of imperative premises holds, and the set Y of factual premises holds, then logic Z dictates that further actions W are imperative.

"I hold X already, and I can convince logic Z of the factual truth of Y, thus I believe W to be imperative."

Even all those complete bastards who disagree with your X can thus come to an agreement with you about the hypothetical as a whole, provided they are epistemically rational. Having isolated the area of disagreement to X, Y, or Z, you can then proceed to argue about it.

Comment author: Garren 02 June 2011 05:14:11PM *  1 point [-]

It's utterly routine for different people to have conflicting beliefs about whether a given act is moral*. And often they can have a useful discussion, at the end of which one or both participants change their beliefs. These conversations can happen without the participants changing their definitions of words like 'moral', and often without them having a clear definition at all.

It may be routine in the sense that it often happens, but not routine in the sense that this is a reliable approach to settling moral differences. Often such disputes are not settled despite extensive discussions and no obvious disagreement about other kinds of facts.

This can be explained if individuals are basing their judgments off differing sets of values that partially overlap. Even if both participants are naively assuming their own set of values is the set of moral values, the fact of overlapping will sometimes mean non-moral considerations which are significant to one's values will also be significant for the other's values. Other times, this won't be the case.

For example, many pro-lifers naively assume that everyone places very high value on all human organisms, so they spend a lot of time arguing that an embryo or fetus is a distinct human organism. Anyone who is undecided or pro-choice who shares this value but wasn't aware of the biological evidence that unborn humans are distinct organisms from their mothers may be swayed by such considerations.

On the other hand, many pro-choicers simply do not place equally high value on all human organisms, without counting other properties like sentience. Or — following Judith Jarvis Thomson in "A Defense of Abortion" — they may place equally high value on all human organisms, but place even greater value on the sort of bodily autonomy denied by laws against abortion.

Morality as the expression of pluralistic value sets (and the hypothetical imperatives which go along with them) is a very neat explanation of the pattern we see of agreement, disagreement, and partially successful deliberation.

Comment author: asr 02 June 2011 07:21:23PM 0 points [-]

I agree with all the claims you're making about morality and about moral discussion. But I don't quite see where any of this is giving me any new insights or tools. Sure, people have different but often overlapping values. I knew that. I think most adults who ever have conversations about morality know that. And we know that without worrying too much about the definition of morality and related words.

But I think everything you've said is also true about personal taste in non moral questions. I and my friends have different but overlapping taste in music, because we have distinct but overlapping set of desiderata for what we listen to. And sometimes, people get convinced to like something they previously didn't. I want a meta-ethics that gives me some comparative advantage in dealing with moral problems, as compared to other sorts of disagreements. I had assumed that lukeprog was trying to say something specifically about morality, not just give a general and informal account of human motivation, values, and preferences.

Thus far, this sequence feels like a lot of buildup and groundwork that is true but mostly not in much dispute and mostly doesn't seem to help me accomplish anything. Perhaps my previous comment should just have been a gentle nudge to lukeprog to get to the point.

Comment author: Garren 02 June 2011 08:00:47PM *  2 points [-]

I want a meta-ethics that gives me some comparative advantage in dealing with moral problems, as compared to other sorts of disagreements.

This may be a case where not getting it wrong is the main point, even if getting it right is a let down.

My own view is quite similar to Luke's and I find it useful when I hear a moral clam to try sorting out how much of the claim is value-expression and how much is about what needs to be done to promote values. Even if you don't agree about values, it still helps to figure out what someone else's fundamental values are and argue that what they're advocating is out of line with their own values. People tend to be mistaken about how to fulfill their own values more than they are about how to fulfill their own taste in music.

Comment author: lukeprog 08 June 2011 06:25:10AM 0 points [-]

People tend to be mistaken about how to fulfill their own values more than they are about how to fulfill their own taste in music.

Yes.

That is why I can interrogate what somebody means by 'ought' and then often show that by their own definition of ought, what they thought they 'ought' to do is not what they 'ought' to do.

Comment author: Peterdjones 02 June 2011 07:27:51PM 0 points [-]

It may be routine in the sense that it often happens, but not routine in the sense that this is a reliable approach to settling moral differences.

Do you know of anything better?

Morality as the expression of pluralistic value sets (and the hypothetical imperatives which go along with them) is a very neat explanation of the pattern we see of agreement, disagreement, and partially successful deliberation.

OTOH, the problem remains that people act on their values, and that one persons actions can affect another person. Pluralistic morality is terrible at translating into a uniform set of rules that all are beholden to.

Comment author: Garren 02 June 2011 07:44:19PM *  0 points [-]

Pluralistic morality is terrible at translating into a uniform set of rules that all are beholden to.

Why is that the test of a metaethical theory rather than the theory which best explains moral discourse? Categorical imperatives — if that's what you're referring to — are one answer to the best explanation of moral discourse, but then we're stuck showing how categorical imperatives can hold...or accepting error theory.

Perhaps 'referring to categorical imperatives' is not the only or even the best explanation of moral discourse. See "The Error in the Error Theory" by Stephen Finlay.

Comment author: Peterdjones 02 June 2011 08:59:12PM 1 point [-]

Why is that the test of a metaethical theory rather than the theory which best explains moral discourse?

Because there is a practical aspect to ethics. Moral discourse involves the idea that people should do the obligatory and refrain from the forbidden. -- irrespective of who they are. That needs explaining as well.

Comment author: Garren 02 June 2011 09:16:22PM 0 points [-]

Moral discourse is about what to do, but it doesn't seem to (at least always) be about what everyone must do for no prior reason.

Comment author: Peterdjones 02 June 2011 10:51:48PM 0 points [-]

Uh-huh. Is that an issue of commission rather than omission? Are people not obligated to refrain from theft murder and rape , their inclinations notwithstanding?

Comment author: Garren 03 June 2011 12:43:55AM *  0 points [-]

If by 'obligated' you mean it's demanded by those who fear being the targets of those actions, yes. Or if you mean exercising restraint may be practically necessary to comply with certain values those actions thwart, yes. Or if you mean doing those things is likely to result in legal penalties, that's often the case.

But if you mean it's some simple fact that we're morally obligated to restrain ourselves from doing certain things, no. Or at least I don't see how that could even possibly be the case, and I already have a theory that explains why people might mistakenly think such a thing is the case (they mistake their own values for facts woven into the universe, so hypothetical imperatives look like categorical imperatives to them).

The 'commission' vs. 'omission' thing is often a matter of wording. Rape can be viewed as omitting to get proper permission, particularly when we're talking about drugging, etc.

Comment author: Peterdjones 05 June 2011 09:32:07PM *  2 points [-]

But if you mean it's some simple fact that we're morally obligated to restrain ourselves from doing certain things, no. Or at least I don't see how that could even possibly be the case, and I already have a theory that explains why people might mistakenly think such a thing is the case (they mistake their own values for facts woven into the universe, so hypothetical imperatives look like categorical imperatives to them).

Well, I have a theory about how it could be the case. Objective morality doesn';t have to be a fact-like thing that is paradoxically indetectable. It could be based on the other source of objectivity: logic and reason. It's an analytical truth that you shouldn't do to others what you wouldn't want done to yourself. You are obliged to be moral so long as you can reason morally in the sense that you will be held responsible.

Comment author: asr 05 June 2011 11:00:37PM 2 points [-]

It's an analytical truth that you shouldn't do to others what you wouldn't want done to yourself.

I'm skeptical that this statement is true, let alone an analytic truth. Different people have different desires. I take the Golden Rule to be a valuable heuristic, but no more than that.

What is your reason for believing that it is true as an absolute rule?

Comment author: BobTheBob 03 June 2011 05:14:28PM 1 point [-]

Just to clarify where you stand on norms: Would you say a person is obligated by facts woven into the universe to believe that 68 + 57 = 125 ? (ie, are we obligated in this sense to believe anything?)

To stick my own neck out: I am a realist about values. I think there are facts about what we ought to believe and do. I think you have to be, to capture mathematical facts. This step taken, there's no further commitment required to get ethical facts. Obviously, though, there are epistemic issues associated with the latter which are not associated with the former.

Morality as the expression of pluralistic value sets (and the hypothetical imperatives which go along with them) is a very neat explanation of the pattern we see of agreement, disagreement, and partially successful deliberation.

Would it be fair to extrapolate this, and say that individual variation in value sets provides a good explanation of the pattern we see of agreement and disagreement between individuals as regards moral values - and possibly in quite different domains as well (politics, aesthetics, gardening)?

You seem to be suggesting meta-ethics aims merely to give a discriptively adequate characterisation of ethical discourse. If so, would you at least grant that many see (roughly) as its goal to give a general characterisation of moral rightness, that we all ought to strive for it?

Comment author: Peterdjones 05 June 2011 09:42:36PM *  2 points [-]

To stick my own neck out: I am a realist about values. I think there are facts about what we ought to believe and do. I think you have to be, to capture mathematical facts.

Facts as in true statements, or facts as in states-of-affiairs?

Comment author: Garren 03 June 2011 06:58:38PM 1 point [-]

Would you say a person is obligated by facts woven into the universe to believe that 68 + 57 = 125 ? (ie, are we obligated in this sense to believe anything?) Would you say a person is obligated by facts woven into the universe to believe that 68 + 57 = 125 ? (ie, are we obligated in this sense to believe anything?)

No, I wouldn't say that. It would be a little odd to say anyone who doesn't hold a belief that 68 + 57 equals 125 is neglecting some cosmic duty. Instead, I would affirm:

In order to hold a mathematically correct belief when considering 68 + 57, we are obligated to believe it equals 125 or some equivalent expression.

(I'm leaving 'mathematically correct' vague so different views on the nature of math are accommodated.)

In other words, the obligation relies on a goal. Or we could say normative answers require questions. Sometimes the implied question is so obvious, it seems strange to bother identifying it.

Would it be fair to extrapolate this, and say that individual variation in value sets provides a good explanation of the pattern we see of agreement and disagreement between individuals as regards moral values - and possibly in quite different domains as well (politics, aesthetics, gardening)?

Yes.

You seem to be suggesting meta-ethics aims merely to give a discriptively adequate characterisation of ethical discourse. If so, would you at least grant that many see (roughly) as its goal to give a general characterisation of moral rightness, that we all ought to strive for it?

I think that's generally the job of normative ethics, and metaethics is a little more open ended than that. I do grant that many people think the point of ethical philosophy in general is to identify categorical imperatives, not give a pluralistic reduction.

Comment author: prase 01 June 2011 10:55:57AM 5 points [-]

I don't understand the terms "world of is" and "world of is not". Does "talking about world of is not" mean "deducing from false assumptions", or is there something more to it? Anyway, "talking about world of is" sounds like the worst kind of continental philosophy babble.

Else, the article is clear, comprehensible and well readable.

Comment author: lukstafi 01 June 2011 11:13:16AM 4 points [-]

While "of is, of is not" didn't hurt my understanding that much, the article would be better off without them.

Comment author: wedrifid 01 June 2011 12:03:29PM 7 points [-]

While "of is, of is not" didn't hurt my understanding that much, the article would be better off without them.

I agree, and also note that the way luke dismisses the "is not" misses much of the point that is trying to be expressed by the phrase. If it is going to be discussed at all it deserves the same kind of parameterizing as 'objective' received.

Comment author: Sniffnoy 01 June 2011 06:52:20PM 2 points [-]

It seems to be essentially a bit of wordplay, in that he uses it to mean two different things. Initially he is contrasting "is/is not" statements with "ought/ought not" statements. Later he talks about things that exist vs. things that don't exist. It doesn't seem to be very helpful though; in the earlier sense, there is no distinction between the "world of is" and the "world of is not". So this seems like it was a bad idea.

Comment author: torekp 04 June 2011 02:00:07AM 0 points [-]

I think there may be a good idea behind it though: view it as a cryptic appeal to Occam's Razor. Various moralists (e.g., Railton, Craig) were shown to be speaking of real things and properties, or of imaginary ones, with their moral language. Why not then hypothesize that all are - albeit less transparently than these two - and do away with the need of a special metaphysics or semantics (or both) for "ought" questions as "opposed" to "is" questions.

Comment author: lukeprog 08 June 2011 07:01:14AM 0 points [-]

Does this make it any clearer?

Comment author: prase 08 June 2011 11:00:43AM 0 points [-]

Yes, it does.

Comment author: Jonathan_Graehl 01 June 2011 05:10:25AM 12 points [-]

Solid and unsurprising.

Comment author: lukeprog 01 June 2011 06:20:29AM 8 points [-]

Thanks, this is exactly the feedback I was hoping to receive. :)

Basically, I want this post and the last one to be where Less Wrongers can send people whenever they appear confused about standard philosophical debates in moral theory: "Wait, stop. Go read lukeprog's article on this and then let me know if you still think the same thing."

Comment author: RichardChappell 03 June 2011 07:23:57PM 4 points [-]

Tangentially:

facts about the well-being of conscious creatures are mind-dependent facts

How so? (Note that a proposition may be in some sense about minds without its truth value being mind-dependent. E.g. "Any experience of red is an experience of colour" is true regardless of what minds exist. I would think the same is true of, e.g., "All else equal, pain is bad for the experiencer.")

Comment author: lukeprog 08 June 2011 06:28:45AM 1 point [-]

I'm borrowing the concept 'well-being of conscious creatures' from Sam Harris, who seems to think of it in terms of mind-dependent facts, perhaps involving (e.g.) brain states we might call 'pain' or 'pleasure'.

Comment author: RichardChappell 08 June 2011 05:32:26PM 1 point [-]

That doesn't really answer my question. Let me try again. There are two things you might mean by "mind dependent".

(1) You might just mean "makes some reference to the mind". So, for example, the necessary truth that "Any experience of red is an experience of colour" would also count as "mind-dependent" in this sense. (This seems a very misleading usage though.)

(2) More naturally, "mind dependent" might be taken to mean that the truth of the claim depends upon certain states of mind actually existing. But "pain is bad for people" (like my example above) does not seem to be mind-dependent in this sense.

Which did you have in mind?

Comment author: lukeprog 08 June 2011 10:35:58PM 0 points [-]

By saying that "facts about the well-being of conscious creatures are mind-dependent facts," I just mean that statements about the well-being of conscious creatures are made true or false by facts abound mind states. A statement about my well-being is mind-dependent in the sense that a statement about my well-being (as I am using the term) is a statement about my brain states. A statement about the distance between my chair and my desk is not a statement about brain states, and would be true or false whether or not our Hubble volume still contained any minds.

Comment author: Oligopsony 04 June 2011 08:25:34PM 0 points [-]

I believe the "facts" in question were synthetic ones ("all else being equal, being set on fire is bad for the person set on fire,") not analytic ones ("all else equal, pain is bad for the experiencer.")

Comment author: RichardChappell 04 June 2011 08:38:03PM *  2 points [-]

It's not analytic that pain is bad. Imagine some crazy soul who thinks that pain is intrinsically good for you. This person is deeply confused, but their error is not linguistic (as if they asserted "bachelors are female"). They could be perfectly competent speakers of the english language, and even logically omniscient. The problem is that such a person is morally incompetent. They have bizarrely mistaken ideas about what things are good (desirable) for people, and this is a substantive (synthetic), not merely analytic, matter.

Perhaps the thought is that contingent (rather than necessary) facts about wellbeing are mind-dependent. That's still not totally obvious to me, but it does at least seem less clearly false than the original (unrestricted) claim.

Comment author: Peterdjones 05 June 2011 05:00:16PM 1 point [-]

The issue is more whether anyone could think pain is good for them themselves. One could imagine a situation where pain receptors connect up to pleasure centers, but then it becomes a moot point as to whether that is actually pain.

Comment author: RichardChappell 05 June 2011 05:56:46PM 3 points [-]

Yes, I was imagining someone who thought that unmitigated pain and suffering was good for everyone, themselves included. Such a person is nuts, but hardly inconceivable.

Comment author: RichardKennaway 05 June 2011 06:30:27PM 0 points [-]

In the not distant past, some surgeons opposed pain-killing medication for post-operative pain, believing that the pain was essential to the healing process.

There's also the reports by patients who have had morphone for pain relief, that the pain is still there, but it takes the hurting out of it.

Comment author: RichardChappell 05 June 2011 07:10:18PM 7 points [-]

Just to clarify: By 'pain' I mean the hurtful aspect of the sensation, not the base sensation that could remain in the absence of its hurting.

In your first paragraph you describe people who take pain to be instrumentally useful in some circumstances, to bring about some other end (e.g. healing) which is itself good. I take no stand on that empirical issue. I'm talking about the crazy normative view that pain is itself (i.e. non-instrumentally) good.

Comment author: poqwku 06 June 2011 07:15:33PM *  2 points [-]

I'm with RichardChappell. Deeming pain intrinsically good is pretty easy, though highly unusual. And deeming pain intrinsically value-neutral is even easier, and not all that unusual. Thus there's nothing incoherent about denying that pain is intrinsically bad.

For example, plenty of people think animal pain doesn't matter, not intrinsically anyway. Perhaps Kant is the most famous example. He thinks cruelty to animals is morally wrong only because it is likely to make us cruel to humans. But the animal pain itself is, Kant thinks, irrelevant and perfectly devoid of any value or disvalue.

Certain Stoics would even say that human pain doesn't matter, not intrinsically anyway. It matters only inasmuch as it causally relates to one's own virtue, and it has no intrinsic relevance to what is good.

If someone went even further, reversing common sense and insisting that pain were intrinsically good, that would be unusual. But it wouldn't be incoherent. Not even close. To invent an example, suppose an extremely credulous religious person were told by their leader that pain is intrinsically good. This true-believer would then be convinced that pain is intrinsically good, and they would try to bring about pain in themselves and in others (so long as they didn't violate any other moral rules endorsed by the leader).

Comment author: Vladimir_Nesov 02 June 2011 02:52:27AM 4 points [-]

The problem is not that there is no way to identify 'good' or 'right' (as used intuitively, without tabooing) with a certain X. The problem is that X is huge and complicated and we don't (yet) have access to its structure.

Strictly speaking, we can exhibit any definition of "good", even one that doesn't make any of the errors you pointed out, and still ask "Is it good?". The criteria for exhibiting a particular definition are ultimately non-rigorous, even if the selected definition is, so we can always examine them further.

Moore's argument might fail in the unintended use case of post-FAI morality not because at some point there might be no more potential for asking the question, but because, as with "Does 2+2 equal 4?", there is a point at which we are certain enough to turn to other projects, even if in principle some uncertainty and lack of clarity in the intended meaning remains. It's not at all clear this will ever happen to morality.

Comment author: JenniferRM 01 June 2011 03:20:57PM *  7 points [-]

When I have serious conversations with thoughtful religious people who have faith but no major theological training, I find it helpful to think of their statements about "God" as being statements about "all worldly optimization processes stronger than me that I don't have time to understand in very much detail like evolution, entropy, economics, democratic politics, organizational dynamics, similar regularities in the structure of the world that science hasn't started analyzing yet, plus many small activist groups throughout history, and a huge number of specific powerful agents silently influencing my life right now like various investors and celebrities, the local chief of police, the local school principal, my employer, my ancestors, and so on".

I can imagine a relatively simple life heuristic, H, that might successfully navigate this vast and bewildering array of optimization pressures in their life and can ask "Does God want you to H". Also, this translation scheme helps me to listen to evangelical radio and learn things from it :-)

I bring this up because it feels to me like you're doing a lot of work to resuscitate ideas from moral philosophy that are significantly helped by "pluralistic reduction" to unpack the ideas into more specific and coherent claims, but you seem to be doing it in a lop-sided way by not unpacking the ideas of "the other side" in a similarly generous manner. Also, to a lesser extent, it seems to be leaving some of "our" ideas unpacked that could probably use some pluralistic reduction but might not look as shiny if unpacked this way.

I guess what I'm trying to say is that "acoustic messenger ferries" in the "ether" seem to me like perfectly adequate placeholder terms if I'm in a conversation with someone whose starting vocabulary uses them as atomic concepts, but if I'm tossing those terms out then "an ideally instrumentally rational and fully informed agent" seems roughly as questionable given how much difficulty people seem to have when using mind-shaped conceptually-atomic entities in their theories.

Do you think my impression of lopsided conceptual unpacking is accurate? If yes, I'm wondering if you could try to introspect on your writing process and try to articulate how you decided which things to unpack and which to leave fuzzy.

Comment author: lukeprog 04 June 2011 11:59:01PM 1 point [-]

I'm not sure what you mean. I unpacked certain concepts as examples, and there are many more we could unpack. Could you say a bit more about what your concern is?

Comment author: RichardChappell 03 June 2011 07:48:44PM 3 points [-]

If we taboo and reduce, then the question of "...but is it good?" is out of place. The reply is: "Yes it is, because I just told you that's what I mean to communicate when I use the word-tool 'good' for this discussion. I'm not here to debate definitions; I'm here to get something done."

I just wanted to flag that a non-reductionist moral realist (like myself) is also "not here to debate definitions". See my post on The Importance of Implications. This is compatible with thinking well of the Open Question Argument, if we think we have an adequate grasp of some fundamental normative concept (be it 'good', 'reason', or 'ought' -- I lean towards 'reason', myself, such that to speak of a person's welfare is just to talk about what a sympathetic party has reason to desire for the person's sake).

Note that if we're right to consider some normative concepts to be conceptually primitive (not analytically reducible to non-normative concepts) then your practice of "tabooing" all normative vocabulary actually has the effect of depriving us of the conceptual tools necessary to even talk about the normative sphere. Consequent talk of people's (incl. God's) desires or dispositions is simply changing the subject, on this way of looking at things.

Out of interest: Will you be arguing anywhere in this sequence against non-reductionist moral realism? Or are you simply assuming its falsity from the start, and exploring the implications from there? (Even the latter, more modest project is of course worth pursing, but I personally would be more interested in the former.) Either way, it'd be good to be clear about this. (You could then skip the silly rhetoric about how what is not "is", must be "is not".)

Comment author: lukeprog 06 June 2011 12:04:26AM 2 points [-]

I'm inclined not to write about moral non-naturalism because I'm writing this stuff for Less Wrong, where most people are physicalists.

What does it mean to you to say that something is a 'fundamental normative concept'? As in... non-reducible to 'is' statements (in the Humean sense)?

Comment author: RichardChappell 06 June 2011 01:14:53AM *  3 points [-]

I was thinking of "fundamental" concepts as those that are most basic, and not reducible to (or built up out of) other, more basic, concepts. I do think that normative concepts are conceptually isolated, i.e. not reducible to non-normative concepts, and that's really the more relevant feature so far as the OQA is concerned. But by 'fundamental normative concept' I meant a normative concept that is not reducible to any other concepts at all. They are the most basic, or bedrock, of our normative concepts.

Comment author: torekp 07 June 2011 01:22:47AM *  3 points [-]

Given the extremely poor access human beings have to the structure of their own concepts, it's dubious that the methods of analytic philosophy can trace those structures. Moreover, concepts typically "cluster together similar things for purposes of inference" ( Yudkowsky ) and thus we can re-structure them in light of new discoveries. Concepts that are connected now might be improved by disconnecting them, or vice versa. It is not at all clear that normative concepts are not included in this (Neurath-style) boat.

Comment author: RichardChappell 06 June 2011 01:26:09AM 2 points [-]

I'm inclined not to write about moral non-naturalism because I'm writing this stuff for Less Wrong, where most people are physicalists

Physicalists could (like Mackie) accept the non-naturalist's account of what it would take for something to be genuinely normative, and then simply deny that there are any such properties in reality. I'm much more sympathetic to this hard-headed "error theory" than to the more weaselly forms of naturalism.

Comment author: lukeprog 06 June 2011 05:42:23AM 2 points [-]

I think many of our normative concepts fail to refer, but that a class of normative concepts often called hypothetical imperatives do refer, thanks to a rather straightforward reduction as given above. Are hypothetical imperatives not 'genuinely normative' in your sense of the phrase? Do you use the term 'normative' when talking about things other than hypothetical imperatives, and do you think those other things successfully refer?

Comment author: RichardChappell 06 June 2011 04:50:45PM 1 point [-]

As I argue elsewhere:

"Hypothetical imperatives thus reveal patterns of normative inheritance. But their highlighted 'means' can't inherit normative status unless the 'end' in question had prior normative worth. A view on which there are only hypothetical imperatives is effectively a form of normative nihilism -- no more productive than an irrigation system without any water to flow through it."

(Earlier in the post explains why hypothetical imperatives aren't reducible to mere empirical statements of a means-ends relationship.)

I tentatively favour non-naturalist realism over non-naturalist error theory, but my purpose in my previous comment was just to flag the latter option as one that physicalists should take (very) seriously.

Comment author: lukeprog 08 June 2011 06:59:25AM *  3 points [-]

Error theory

You know this, but for the benefit of others: Roughly, error theory consists of two steps. As Finlay puts it:

(1) Presupposition: moral judgments involve a particular kind of presupposition which is essential to their status as moral; (2) Error: this presupposition is irreconcilable with the way things are

Given my view of conceptual analysis, it shouldn't be surprising that I'm not confident of some error theorists' assertion of step 1. Is a presupposition of moral absolutism 'essential' to a judgment's status as a 'moral' judgment? Is a presuppositional of motivational internalism 'essential' to a judgment's status as a 'moral' judgment? I don't know. Moral discourse (unlike carbon discourse) is so confused that I'm not too interested to assert one fine boundary line around moral terms over another.

So if someone thinks a presupposition of supernaturalism is 'essential' to a judgment's status as a 'moral' judgment, then I will claim that supernaturalism is false. But this doesn't make me an error theorist because I don't necessarily agree that a presupposition of supernaturalism is 'essential' to a judgment's status as a 'moral' judgment. I reject step 1 of error theory in this case.

Likewise, if someone thinks a presupposition of moral absolutism or motivational internalism is essential to a judgment's status as a 'moral' judgment, I'll be happy to deny both moral absolutism and motivational internalism, but I wouldn't call myself an error theorist because I reject the claim that moral judgments (by definition, by conceptual analysis) necessarily presuppose moral absolutism or motivational internalism.

But hey, if you convince me that the presumption of motivational internalism in moral discourse is so widespread that talking about 'morality' without it would be like using the term 'phlogiston' to talk about oxygen, then I'll be happy to call myself an error theorist, though none of my anticipations will have changed.

Hypothetical imperatives

I'll reply to a passage from your post on hypothetical imperatives. My reply won't make sense to those who haven't read it:

When we affirm the first premise as a mere hypothetical imperative, we mean it in a sense that does not validate such an inference. We might add, "But of course you shouldn't want to torture children, and so you shouldn't take the means to this atrocious end either."

I think this is because 'should' is being used in different senses. The real modus ponens is:

  1. If you want to torture children, you should_ToTortureChildren volunteer as a babysitter.
  2. You want to torture children.
  3. Therefore, you should_ToTortureChildren volunteer as a babysitter.

Or at least, that is plausibly what some people mean when they assert what looks like a hypothetical imperative. Doubtless, others will appear to be meaning something else if pressed by interrogation.

Now, to respond to Sidgwick:

When (e.g.) a physician says, "If you wish to be healthy you ought to rise early," this is not the same thing as saying "early rising is an indispensable condition of the attainment of health." This latter proposition expresses the relation of physiological facts on which the former is founded; but it is not merely this relation of facts that the word 'ought' imports: it also implies the unreasonableness of adopting an end and refusing to adopt the means indispensable to its attainment.

I could capture this 'unreasonableness' by simply clarifying that from the standpoint of Bayesian rationality, it would be somewhat irrational to expect good health despite not rising early (or so the doctor claims).

But again, I'm not too keen to play the definitions game. If you state hypothetical imperatives with more intuitions about the meaning of hypothetical imperatives than I do, then you are free to explain what you mean by hypothetical imperatives and then show how they fit into the physical world. If you can't show how they fit into the world, then you're talking about something that doesn't exist, or else we'll have to replay the physicalism vs. non-physicalism debate, which is another topic.

Right?

Comment author: RichardChappell 08 June 2011 06:09:23PM *  3 points [-]

Thanks for this reply. I share your sense that the word 'moral' is unhelpfully ambiguous, which is why I prefer to focus on the more general concept of the normative. I'm certainly not going to stipulate that motivational internalism is true of the normative, though it does seem plausible that there's something irrational about someone who acknowledges that they really ought (all things considered) to phi and yet fails to do so. (I don't doubt that it's possible for someone to form the judgment without any corresponding motivation though, as it's always possible for people to be irrational!)

I trust that we all have a decent pre-theoretic grasp of normativity (or "ought-ness"). The question then is whether this phenomenon that we have in mind (i) is reducible to some physical property, and (ii) actually exists.

Error theory (answering 'no' and 'no' to the two questions above) seems the most natural position for the physicalist. And it sounds like you may be happy to agree that you're really an error theorist about normativity (as I mean it). But then I'm puzzled by what you take yourself to be doing in this series. Why even use moral/normative vocabulary at all, rather than just talking about the underlying natural properties that you really have in mind?

P.S. What work is the antecedent doing in your conditional?

If you want to torture children, you should_ToTortureChildren volunteer as a babysitter.

Why do you even need the modus ponens? Assuming that "should_ToTortureChildren" just means "what follows is an effective means to torturing children", then isn't the consequent just plain true regardless of what you want? (Perhaps only someone with the relevant desire will be interested in this means-ends fact, but that's true of many unconditional facts.)

Comment author: lukeprog 08 June 2011 10:59:53PM *  2 points [-]

there's something irrational about someone who acknowledges that they really ought (all things considered) to phi and yet fails to do so. (I don't doubt that it's possible for someone to form the judgment without any corresponding motivation though, as it's always possible for people to be irrational!)

Right.

And it sounds like you may be happy to agree that you're really an error theorist about normativity (as I mean it).

Unfortunately, I don't think I'm clear about what you mean by normativity. The only source of normativity I think exists is the hypothetical imperative, which can be reduced to physics by straightforward methods such as the one I used in the original post. I'm not an error theorist about that kind of normativity.

Why even use moral/normative vocabulary at all, rather than just talking about the underlying natural properties that you really have in mind?

This is a good question. Truly, I want to get away from moral vocabulary, and be careful around normative vocabulary. But people already think about these topics in moral and normative vocabulary, which is why I'm trying to solve or dissolve (in this post and its predecessor) some of the usual 'problems' in this space of questions.

After that's done, I don't think it will be most helpful to use moral language. This is evident in the fact that in 15 episodes of my 'morality podcast' I've used almost no moral language at all.

What work is the antecedent doing in your conditional?

Not much, really. I wasn't using the modus ponens to present an argument, but to unpack one interpretation of (some) 'should' discourse. Normative language, like many other kinds of language, is (when used correctly) merely a shortcut for saying something else. I can imagine a language that has no normative language at all. In that language we couldn't say things like "If you want to torture children, you should volunteer as a babysitter" but we could say things like "If you volunteer as a babysitter you will have more opportunities to torture children." The way I'm parsing 'should' in the first sentence, nothing is lost by this translation.

Of course, people use 'should' in a variety of ways, some of which translate into claims about things reducible to physics, others of which translate into claims about things non-reducible to physics, while still others don't seem to translate into cognitive statements at all.

Comment author: RichardChappell 08 June 2011 11:48:09PM 3 points [-]

Thanks, this is helpful. I'm interested in your use of the phrase "source of normativity" in:

The only source of normativity I think exists is the hypothetical imperative

This makes it sound like there's a new thing, normativity, that arises from some other thing (e.g. desires, or means/ends relationships). That's a very realist way of talking.

I take it that what you really want to say something more like, "The only kind of 'normativity'-talk that's naturalistically reducible and hence possibly true is hypothetical imperatives -- when these are understood to mean nothing more than that a certain means-end relation holds." Is that right?

I'd then understand you as an error theorist, since "being a means-end relationship", like "being red", is not even in the same ballpark as what I mean by "being normative". (It might sometimes have normative importance, but as we learn from Parfit, that's a very different thing.)

Comment author: lukeprog 09 June 2011 12:03:44AM 3 points [-]

My thought process on sources of normativity looks something like this:

People claim all sorts of justifications for 'ought' statements (aka normative statements). Some justify ought statements with respect to natural law or divine commands or non-natural normative properties or categorical imperatives. But those things don't exist. The only justification of normative language that fits in my model of the universe is when people use 'ought' language as some kind of hypothetical imperative, which can be translated into a claim about things reducible to physics. There are many varieties of this. Many uses of 'ought' terms can be translated into claims about things reducible to physics. If somebody uses 'ought' terms to make claims about things not reducible to physics, then I am suspicious of the warrant for those claims. When interrogating about such warrants, I usually find that the only evidences on offer are pieces of folk wisdom, intuitions, and conventional linguistic practice.

Comment author: poqwku 06 June 2011 07:33:49PM *  0 points [-]

I don't think hypothetical imperatives can be reduced. The if-ought of a hypothetical imperative is a full-blooded normative claim. But you can't reduce that to a simple if-then about cause and effect.

To see why, consider a nihilist about oughts. She recognizes the causal connections between calorie consumption/burning and weight loss. But she doesn't accept any claim about what people ought to do, even hypothetical imperatives about people who desire weight loss. This seems perfectly coherent: she accepts causal claims, but not normative claims, and there's no contradiction or incoherence there. But this means the causal claims she accepts are not conceptually equivalent to the normative claims she rejects.

For another way to see why, consider the causal claim "less calorie consumption and more calorie burning leads to weight loss". This causal claim points in no normative direction. It doesn't recommend anything, or register any approval, or send any positive or negative messages. Of course, we can take it in one direction or another, but only by combining it with separate normative claims:

  1. Less calorie consumption and more calorie burning leads to weight loss.
  2. People ought to take causally efficacious steps to satisfy their desires.
  3. Therefore, if you desire to lose weight, you ought to consume less calories and burn more calories.

Premise 2 is what provides the normativity. It points us in the direction of satisfying desires. But we could easily take things in the opposite direction.

  1. Less calorie consumption and more calorie burning leads to weight loss.
  2. People ought to take causally efficacious steps to frustrate their desires.
  3. Therefore, if you desire to lose weight, you ought to consume more calories and burn less calories.

Again, premise 2 is what provides the normativity. But it points us in the opposite direction, viz. the direction of frustrating desires.

So it's pretty clear that premise 1 has no normativity in it. It can't be reduced to either of the two 3's. For we cannot arrive at a 3 without a 2.

Comment author: torekp 07 June 2011 12:51:37AM *  0 points [-]

I think stating premise 2 is a little odd. It is a bit deja "Tortoise and Achilles" all over again. If there's a norm hiding around here, it's an "ought" portrayed by the desire.

Second, conceptual analysis (or conceptual equivalence) is not necessary for reduction. Look at reduction in the sciences for examples.

Comment author: poqwku 07 June 2011 07:14:15PM *  1 point [-]

Well, I'll acknowledge that you could change premise 2 into an inference rule. But notice that you could change either premise 2—the pro-desire-satisfaction one and the pro-desire-frustration one—into an inference rule. Indeed, you could change any normative claim into an inference rule: you could change "people who want to have gay sex ought to go see a trained Baptist minister to get cured" into an inference rule, and then validly go from "I want to have gay sex" to "I ought to go see a trained Baptist minister to get cured". So from the fact that premise 2 could be changed into an inference rule, I don't think anything follows that might jeopardize its status as a full-blooded normative claim.

On the second point, I thought lukeprog was discussing direct conceptual reduction. But if he wants to provide hypothetical imperatives with a synthetic reduction, he'll need a theory of reference capable of explaining why the normative claim turns out to make reference to (and have its truth-conditions provided by) simple causal facts. And on this score, I think hypothetical imperatives and categorical moral imperatives are on an equal footing: since reductionist moral realists have a hard time with synthetic reductions, I would expect reductionist 'instrumental realists' to have a hard time as well.

Comment author: torekp 08 June 2011 10:16:37PM *  0 points [-]

For what it's worth, I think what's really being inferred by the advice-giver is:

  • 1 Granting your (advisee's) starting point, you ought to lose weight.
  • 2 Less calorie consumption and more calorie burning leads to weight loss.
  • 3 Therefore you ought to consume less and burn more calories.

The advisee's desire portrays the starting-point as a truth.

Comment author: poqwku 11 June 2011 12:05:13AM *  0 points [-]

Perhaps so, but then the normativity stems from premise 1, leaving premise 2 as non-normative as ever. But the question is whether premise 2 could be a plausible reduction basis for normative claims.

Comment author: zefreak 03 June 2011 07:07:15PM *  3 points [-]

I think you are incorrect with regards to Hume's is-ought gap, although I find its relevance to be somewhat overstated. A hypothetical imperative such as your example relies on an equivocation between 'ought' as (1) a normative injunction and (2) conveying a possible causal pathway from here to there.

-

Here is the incorrect syllogism:

Premise 1: A desires C (is)

Premise 2: B will produce C (is)

Conclusion: A ought to do B (ought)

-

There is a hidden normative premise that is often ignored. It is

Premise 3: A should obtain its desires. (ought)

-

The correct syllogism would then be:

Premise 1 (is): A desires C

Premise 2 (is): B will produce C

Premise 3 (ought): A ought to obtain its desires.

Conclusion: A ought to do B (ought)

-

The necessity of Premise 3 is made clear by use of an admittedly extreme example:

P1: Hitler wants to kill a great number of people

P2: Zyklon B will kill a great number of people

C1: Hitler ought to use Zyklon B to kill a great number of people

While the conclusion is derived from the premises using definition (2) of the word 'ought', few would express it as a normative recommendation.

-

Hume's fact/value dichotomy remains valid. A normative conclusion can only be validly deduced from a group of premises including at least one which is itself normative.

Comment author: Peterdjones 01 June 2011 05:04:03PM *  5 points [-]

Or, perhaps someone has a moral reductionism in mind during a particular use of 'ought' language. Perhaps by "You ought to be more forgiving" they really mean "If you are more forgiving, this is likely to increase the amount of pleasure in the world."

As you can see, it is not hard to bridge the is-ought gap.

I don't think it is impossible, but it is harder than you are making out. The examples given are not complete syllogisms, or other logical forms. It is easy to validly derive an ought form an is: you start with the factual statement and then invoke a bridging principle of the form:

if <is statement> then <ought statement>

However, the argument is not sound unless the bridging statement is true. But the bridging statement is itself an derivation of an ought from an is, so there is a kind of circularity there. You are assuming that the ought-from-is problem has been solved in order to solve it.

As I said, I don't think the situation is hopeless. The bridging premise is not exactly the same thing as a moral argument: it is usually more of a general statement along the lines of "if X increases well being, it should be done". That provides some scope for an analytical justification of bridging principles.

Comment author: Vladimir_Nesov 02 June 2011 03:10:48AM *  4 points [-]

That's why we need to decode the cognitive algorithms that generate our questions about value and morality. ... So how can the Empathic Metaethicist answer Alex's question? We don't know the details yet. For example, we don't have a completed cognitive neuroscience.

Assume you have a complete knowledge of all details of the way human brain works, and a detailed trace of the sequence of neurological events that leads people to ask moral questions. Then what?

My only guess is that you look this trace over using your current moral judgment, and decide that you expect that changing certain things in the algorithm will make the judgments of this brain better. But this is not a FAI-grade tool for defining morality (unless we have to go the uploads-driven way, in which case you just gradually and manually improve humans for a very long time).

Comment author: lukeprog 08 June 2011 07:03:49AM 1 point [-]

Yes, a completed cognitive neuroscience would certainly not be sufficient for defining the motivational system of an FAI.

Comment author: lukeprog 01 June 2011 01:08:57AM *  4 points [-]
Comment author: Jack 21 August 2011 06:36:58PM 0 points [-]

(Replying to the comment instead of the post to be sure you'll see this)

In "The Is-Ought Gap" you conclude

Either our intended meaning of 'ought' refers (eventually) to the world of math and physics (in which case the is-ought gap is bridged), or else it doesn't (in which case it fails to refer).

I have a lot to say about this particular issue but I'm not sure if you think you've exhausted the issue in this post or if you plan to come back to it. Just to begin with, though, I hope you're aware that the two reasonable camps that take the gap most seriously would both agree with this conclusion. The issue is exactly this: we don't think 'ought' refers.

Comment author: Emile 02 June 2011 12:34:23PM 2 points [-]

Here are some common uses of the objective/subjective distinction in ethics:

  • Moral facts are objective1 if they are made true or false by mind-independent facts, otherwise they are subjective1.
  • Moral facts are objective2 if they are made true or false by facts independent of the opinions of sentient beings, otherwise they are subjective2.
  • Moral facts are objective3 if they are made true or false by facts independent of the opinions of humans, otherwise they are subjective3.

Hmm, that doesn't cover the way I understand "objective" and "subjective" - I see them as referring to whether the answer to a question varies from one person to another, i.e. whether they are function of the person speaking. "Is that play interesting?" is subjective4, "Does that play follow the three unities of classical drama?" is objective4 - so to match your pattern it would be something like "Moral facts are objective4 if they are made true or false by facts independent of any single human, otherwise they are subjective4".

i.e. it seems to me that some things are considered "objective" if they are merely social customs that may not hold in another society in another age.

But my thinking around this isn't very clear, I'm mostly reacting to the fact that none of the definitions you listed seemed to fit the way I understood the words.

Comment author: torekp 04 June 2011 02:46:43AM 2 points [-]

I agree that a very common meaning of objective/subjective is along your lines. In metaethics, it seems the most discussed. But I'd suggest function of the speaker's beliefs and attitudes, not just function of the speaker. My left hand has five fingers: this truth is a function of the speaker, but if we want to communicate effectively in English, we'd better call this fact objective.

Comment author: nshepperd 08 June 2011 11:50:45AM *  1 point [-]

It seems to me that this subjective/objective difference can be somewhat dissolved by noticing (or, if you like, postulating) that "Is that play interesting?" is a different question when asked of Alice than when asked of Bob. The real question is, probably, in one case "Is that play interesting to Alice?" and in the other case "Is that play interesting to Bob?".

In sentences like "Do you like vanilla ice cream?" it's more explicit and clear that the meaning of "you" in it must be inferred from context (specifically, the identity of the questionee).

I would postulate that in general a "question" is directly about facts about the universe or mathematics (hopefully grounded in anticipation of sensory experience), but the sentence the question is asked in may contain ambiguous words which require context to interpret. And further, if the sentence together with the context is unambiguous, it indicates a "real question".

In either case of the first example, the question has a single truth value which doesn't change depending who you ask, though Alice and Bob may react differently to the sound waves (or light patterns) that are understood as "Is that play interesting?".

Perhaps the sentences usually called "subjective4 questions" would be those containing such ambiguous words as "interesting" or "sexy", in regard to which people are likely to suffer from the mind projection fallacy... I don't know.

Comment author: Peterdjones 08 June 2011 01:37:13PM *  0 points [-]

It seems to me that this subjective/objective difference can be somewhat dissolved by noticing (or, if you like, postulating) that "Is that play interesting?" is a different question when asked of Alice than when asked of Bob. The real question is, probably, in one case "Is that play interesting to Alice?" and in the other case "Is that play interesting to Bob?".

I don't see why that would count as a dissolution. The subjective is defined as varying with individuals and the objective is defined as not so varying. All that your restatements as ".."to Alice", "..to Bob" do is make that dependence explicit.

Perhaps the sentences usually called "subjective4 questions" would be those containing such ambiguous words as "interesting" or "sexy",

These words look ambiguous if you expect them to have a single value. They can be unambiguous if they have well defined values for different people. Subjective-ness isn't ambiguity.

Comment author: nshepperd 09 June 2011 10:30:35AM 0 points [-]

I guess I just don't think it's interesting or in any way special that there are questions that are usually asked by including non-verbal information, or that there are words that refer to "the recipient of this message".

Wrt "ambiguity" all I mean is that you don't know what the speaker intended the word to refer to until you know who they were talking to. Steve said "Is that play interesting?" to Alice, implying he wants to know whether Alice found the play interesting. Responding with Bob's opinion of the play would be unhelpful, which is why you need the context. Maybe "ambiguous" isn't the best word for that. Whatever.

Comment author: Peterdjones 09 June 2011 12:25:46PM *  0 points [-]

I guess I just don't think it's interesting or in any way special that there are questions that are usually asked by including non-verbal information, or that there are words that refer to "the recipient of this message".

That would be OK if a) there were a clear distinction between the two categories and/or b) nothing much rode on the distinction.

But neither is the case wrt morallity. a) We don't have a "usual" practices with regard to moral language. Committed objectivists speak one way, subjectivists another, and many others are undecided b) it is hard to coneive of anything more important than morality -- and the two ways of speaking mean something different. Alice can't wish Bob to be punished just for doing something that's wrong-for-Alice.

Comment author: Peterdjones 08 June 2011 10:51:12AM 1 point [-]

I see them as referring to whether the answer to a question varies from one person to another, i.e. whether they are function of the person speaking. "Is that play interesting?" is subjective4, "

Answers can vary because people make mistakes about objective issues. You need to specify ideal agents to define objectivity, or to define subjectivity as answers that properly vary with individuals, ie the individual has the last word on their favourite flavour of ice cream.

Comment author: lukeprog 08 June 2011 07:13:49AM 0 points [-]

Certainly, there are far more than 3 uses of the objective/subjective distinction! Check for footnote for a pointer to others.

Comment author: Vladimir_Nesov 02 June 2011 02:17:01AM *  2 points [-]

Moral facts are objective3 if they are made true or false by facts independent of the opinions of humans, otherwise they are subjective3.

Are the words of this comment subjective3 (if we drop the specifier "moral" for a moment, given that this idea is not defined in this context)? They are determined by my reasoning, but they also obey the laws of physics. The notion of "independent" is not easy to make precise.

Comment author: lukeprog 08 June 2011 07:08:03AM 0 points [-]

Right. It would take a lot of unpacking to make precise the term 'mind-independent.'

Comment author: wedrifid 02 June 2011 11:43:26AM *  0 points [-]

Are the words of this comment subjective3 (if we drop the specifier "moral" for a moment, given that this idea is not defined in this context)? They are determined by my reasoning, but they also obey the laws of physics. The notion of "independent" is not easy to make precise.

I had similar thoughts when reading that passage.

There is (also) a certain sense in which the concept 'objectively subjective' is relevant here.

Comment author: Peterdjones 01 June 2011 06:37:15PM 5 points [-]

the Austere Metaethicist replies:

"Tell me what you mean by 'right', and I will tell you what is the right thing to do."

That is of course, not what is right, but what she thinks is right. So far, so subjective.

You may not know what you mean by 'right.' But let's not stop there. Here, let me come alongside you and help decode the cognitive algorithms that generated your question in the first place, and then we'll be able to answer your question. Then we can tell you what the right thing to do is.

Again, that is not the right thing, that is just what she thinks. An Objective metaethicist could answer the question what is right.

But moral terms and value terms are about what we want.

No: they are value terms about what we should want and be and do.

And the "we" is important here. Your metaethicists are like therapsists or life coaches or personal shoppers who advise people how to make their individual lives spiffier. But moral action is not solipsistic: moral choices affect other people. That's why we can't stop at "whatever you think is right is right". I don't want one of your metaethicists telling my neighbour how to be a better serial killer.

Comment author: Manfred 01 June 2011 10:03:19PM *  1 point [-]

I think you're missing the point of why there isn't a universal code of ethics. If two people disagree about the definition of a word, the way forward isn't to jump into the argument on one side or the other. The way forward is to stop using that word that way. This is subjective with respect to the word (if you don't also specify how you're defining it) but we stopped using that word that way anyhow - it's objective with respect to the world, and that's what's important to how people act.

Comment author: Peterdjones 01 June 2011 10:46:06PM 0 points [-]

If two people disagree about the definition of a word, the way forward isn't to jump into the argument on one side or the other. The way forward is to stop using that word that way.

If someone uses "cat" to mean "animal that barks", should everyone then stop using "cat"?

This is subjective with respect to the word (if you don't also specify how you're defining it) but we stopped using that word that way anyhow - it's objective with respect to the world, and that's what's important to how people act.

I can't make any sense of that.

Comment author: FAWS 02 June 2011 12:52:56AM 0 points [-]

If someone uses "cat" to mean "animal that barks", should everyone then stop using "cat"?

In conversations with that particular person, assuming they can't easily be persuaded to change their usage? Yes, definitely.

Comment author: Peterdjones 02 June 2011 01:05:36AM 0 points [-]

That's hardly an optimal outcome. They are making a mistake, although it seems no one wants to admit that.

Comment author: FAWS 02 June 2011 11:27:47AM 0 points [-]

Obviously the "optimal outcome" would be the easy persuasion I mentioned. Do you think someone misusing that word justifies arbitrary high effort in persuasion, or drastic measures?

Comment author: Manfred 02 June 2011 12:07:51AM 1 point [-]

If someone uses "cat" to mean "animal that barks", should everyone then stop using "cat"?

You're right, it's more complicated. It seems like the solution here is to make word choice a coordination problem, communication being a major goal of language - if a million people use it one way and one person uses it the other way, the one should say "an animal that barks." On the other hand if everyone has the same several definitions for a word, like "sound," then splitting up the word when necessary improves communication.

This is subjective with respect to the word (if you don't also specify how you're defining it) but we stopped using that word that way anyhow - it's objective with respect to the world, and that's what's important to how people act.

I can't make any sense of that.

You complain that letting people specify what they mean by "right" makes "right" subjective where people diverge. But this doesn't make the communication subjective if people replace "right" by an objective criterion for the world, so the bad stuff associated with just drifting off into subjectivity doesn't happen.

Although I guess you could be saying that "right" being subjective is inherently bad. But I would suggest that you're thinking about right_Peter, which is still objective.

Comment author: Peterdjones 02 June 2011 12:37:01AM *  0 points [-]

If everyone has their own notion of right, we still have the Bad Thing that an action can only be allowed of forbidden, not Peter-allowed and Manfred-forbidden.

Comment author: Manfred 02 June 2011 12:59:29AM 0 points [-]

? So it's impossible for two people to rationally disagree about whether or not an action is forbidden if the external state of the world is the same? I see no reason why "forbidden" in the moral sense should be objective.

Comment author: Peterdjones 02 June 2011 01:02:16AM 0 points [-]

So it's impossible for two people to rationally disagree about whether or not an action is forbidden if the external state of the world is the same?

If they disagree and they are both being rational, where's the objectivity?

I see no reason why "forbidden" in the moral sense should be objective.

Try explaining to someone that something they like should be forbidden because you don't like it.

Comment author: Manfred 02 June 2011 01:56:14AM 0 points [-]

So it's impossible for two people to rationally disagree about whether or not an action is forbidden if the external state of the world is the same?

If they disagree and they are both being rational, where's the objectivity?

I agree, it doesn't look like there is much in this concept.

I see no reason why "forbidden" in the moral sense should be objective.

Try explaining to someone that something they like should be forbidden because you don't like it.

Okay. "If you don't stop, I will shoot you."

But seriously, WTF? Is that supposed to be an argument that if something is morally forbidden to one person it should be the same for another person?

Comment author: Peterdjones 02 June 2011 02:23:11AM -1 points [-]

I agree, it doesn't look like there is much [objectivity] in this concept.

If you don't go looking for it, you won't find it. As is so ofen the case on LW, that door has been shut without even trying to see what is behind it.

I see no reason why "forbidden" in the moral sense should be objective

Try explaining to someone that something they like should be forbidden because you don't like it.

Okay. "If you don't stop, I will shoot you."

Arbitrary rules enforced with threats of violence is not an optimal outcome for me.

If you have an option other than

a) subjective laissez faire where serial killers are allowed to do their own thing

or

b) Tyranny

I'd be glad to hear it. I know I have.

But seriously, WTF? Is that supposed to be an argument that if something is morally forbidden to one person it should be the same for another person?

Of course. No one should murder. I'm surprised you find that surprising.

Comment author: Nornagest 02 June 2011 06:42:28AM *  0 points [-]

Arbitrary rules enforced with threats of violence is not an optimal outcome for me. If you have an option other than

a) subjective laissez faire where serial killers are allowed to do their own thing

or

b) Tyranny

I'd be glad to hear it. I know I have.

Assuming you're generalized that properly and aren't seriously arguing the most egregious false dichotomy I've seen in weeks, I'm afraid that condemning the set of ethics based on social or personal consequences as "tyranny" amounts to dismissing an entire school of thought on aesthetic grounds. Forgive me if I don't find such a thing particularly convincing.

Comment author: Manfred 02 June 2011 05:04:56AM 0 points [-]

If you don't go looking for it, you won't find it.

Rather than chastising me, why not explain how "forbidden" is objective?

Re: shooting people: it was a joke. The WTF was not with respect to shooting people. It was because your demand was a non sequitur.

Comment author: Will_Sawin 01 June 2011 10:19:12PM 0 points [-]

Why do you think it's the definition of the word that's at issue?

Comment author: Manfred 01 June 2011 10:53:38PM 0 points [-]

Because it is possible for people to disagree about whether something is right or wrong without disagreeing about the state of the world.

Comment author: Will_Sawin 01 June 2011 11:24:27PM 1 point [-]

Why must all disagreements be disagreements about the state of the world?

It seems to me like there are two kinds of disagreements, positive disagreements about the state of the world, and normative disagreements about the proper state of the world.

Nothing blows up when I believe that.

Knowing the fact that you just stated does not make fighting wars over morality seem less reasonable. In fact, it makes them seem more reasonable. Do you disagree? Do you think it makes sense to fight wars over a definition?

Comment author: Manfred 02 June 2011 12:10:16AM 0 points [-]

It seems to me like there are two kinds of disagreements, positive disagreements about the state of the world, and normative disagreements about the proper state of the world.

Sure. But if people start arguing over what's right, they should argue over the proper state of the world, not over what's "right."

Comment author: Peterdjones 02 June 2011 12:11:26AM 0 points [-]

I don't see much difference between "right" and "proper".

Comment author: torekp 04 June 2011 02:10:47AM 0 points [-]

I agree that Luke's approach has at times seemed implausibly individualistic. Moral reasoning is interpersonal from the get-go.

Comment author: nshepperd 02 June 2011 02:26:18AM 0 points [-]

You're missing the point. The empathic metaethicist is trying to figure out what she means by 'right'. Assuming she's a well-adjusted human being, that's probably the same as what you mean by 'right', so with any luck we'll work out what you mean by 'right' as well (and hence, what "is right"). But we're not asking Alex what she thinks peterdjones.getMeaning("right").getExtension() is.

Comment author: Peterdjones 02 June 2011 02:31:25AM 0 points [-]

That isn't a good theoretical argument that "right" has only a subjective definition, and it isn't practically as good as being able make individual notions of moral rightness more correct, where they need fixing.

Comment author: nshepperd 02 June 2011 03:50:54AM 1 point [-]

Whatever you mean by "only a subjective definition", I'm probably not trying to argue that.

Do you think you mean something other than what is right when you say "right"? If not, then replace "Alex" with "Peterdjones". Do you still think the empathic metaethicist is going to get the wrong answer when they try to figure out what you mean by "right"?

Comment author: Will_Sawin 01 June 2011 01:41:05AM *  4 points [-]

Consider this dialog:

Student: "Wise master, what ought I do?"

Wise master: "You ought to help the poor by giving 50% of your income to efficient charity and supporting the European-style welfare state."

Student: "Alright."

*student runs off and gives 50% of his or her income to efficient charity and supports the European-style welfare state

This dialog rings true as a fact about ought statements - once we become convinced of them, they do and should constrain our behavior.

But my dialogs and your dialogs contradict each other! Because if "ought" determines our behavior, and we can define what "ought" means, then we can define proper behavior into existence - a construction as absurd as Descartes defining God into existence or Plato defining man as both a hairless featherless biped and a mortal.

We must give up one, and I say give up yours. "ought" is one of those words that we are not free to define - it has a single meaning. Look to its consequences, not its causes.

Comment author: lukeprog 01 June 2011 02:10:38AM *  2 points [-]

I'm not sure I understand. Are you saying that we are not free to stipulate definitions for the word-tools we use (when it comes to morality), because you have a conceptual intuition in favor of motivational internalism for the use of 'ought' terms?

Comment author: Will_Sawin 01 June 2011 02:43:59AM *  2 points [-]

Wikipedia defines motivational internalism as the belief that:

there is an internal, necessary connection between one's conviction that X ought to be done and one's motivation to do X.

I want to view this as a morally necessary connection. One should do what one ought to do, and this serves as the definition of "ought".

You will note that I am using circular definitions. That is because I can't define "should" except in terms of things that have a hidden "should" in there. But I am trying to access the part of you that understands what I am saying.

The useful analogue is this:

modus ponens: "If you know 'A', and you know 'If A, then B", then you know B"

It's a circular definition getting at something which you can't put into words. I would be wrong to define "If-then" as something else, like maybe "If A, then B" means "75% of elephants with A written on them believe B" because it's already defined.

Does that make any sense?

Comment author: lukeprog 01 June 2011 02:46:32PM *  1 point [-]

Unfortunately, I still don't follow you. Or at least, the only interpretations I've come up with look so obviously false that I resist attributing them to you. Maybe I can grok your disagreement from another angle. Let me try to pinpoint where we disagree. I hope you'll have some time to approach mutual understanding on this issue. When Will Sawin disagrees with me, I pay attention.

Do you agree that there are many words X such that X is used by different humans to mean slightly different things?

Do you agree that there are many words X such that different humans have different intuitions about the exact extension of X, especially in bizarre sci-fi hypothetical scenarios?

Do you agree that many humans use imperative terms like 'ought' and 'should' to communicate particular meanings, with these meanings often being stipulated within the context of a certain community?

I'll stop there for now.

Comment author: Will_Sawin 01 June 2011 03:42:32PM *  1 point [-]

Thanks. I'm thinking of doing a post on the discussion section where I can explain where my intuitions come from in more detail.

For your questions:

Yes.

Yes.

I don't really know what the third question means. It seems like the primary use of "ought" and "should" is as part of an attempt to convince people to do what you say they should do. I would say that is the meaning being communicated. There are various ways this could be within the context of a community. Are you saying that you're only trying to convince members of that community?

Comment author: lukeprog 04 June 2011 09:19:07PM 0 points [-]

Note: I'm planning to come back to this discussion in a few days. Recently my time has been swamped running SI's summer minicamp.

Comment author: Will_Sawin 04 June 2011 09:34:43PM 3 points [-]

I may also write something which expresses my ideas in a new, more concise and clear form.

Comment author: lukeprog 05 June 2011 11:20:39PM 0 points [-]

I think that would be the most efficient thing to do. For now, I'll wait on that.

Comment author: Will_Sawin 09 June 2011 06:59:17PM 1 point [-]

If you haven't noticed, I just made that post.

Comment author: lukeprog 13 June 2011 05:58:23AM 0 points [-]

Any response to this?

Comment author: lukeprog 10 June 2011 04:55:31PM 0 points [-]

Excellent. I'm busy the next few days, but I'll respond when I can, on that thread.

Comment author: steven0461 01 June 2011 03:43:52AM *  1 point [-]

That is because I can't define "should" except in terms of things that don't have a hidden "should" in there.

I think you meant to leave out either the "except" or the "don't"?

Comment author: Will_Sawin 01 June 2011 10:57:13AM 0 points [-]

Correct.

Comment author: Normal_Anomaly 01 June 2011 02:09:34AM 1 point [-]

Your dialogue looks similar to the one about losing weight above. I can define proper behavior given my terminal values. If I want to lose weight, I should eat less. Upon learning this fact, I start eating less. My values and some facts about the world are sufficient to determine my proper behavior. "Defining my behavior into existence" seems no more absurd to me than defining the rational action using a decision theory.

I'm not sure I've explained myself very clearly here. Please advise on what, if anything, that I'm saying is hard to understand.

Comment author: Will_Sawin 01 June 2011 02:47:33AM -1 points [-]

If it is the case that you should do what you want, yes.

If you want to punch babies, then you should not punch babies. (x)

If you should lose weight, then you should eat less.

Proper values and some facts about the world are sufficient to determine proper behavior.

What are proper values? Well, they're the kind of values that determine proper behavior.

x: Saying this requirems me to know a moral fact. This moral fact is a consequence of an assumption I made about the true nature of reality. But to assume is to stoop lower than to define.

Comment author: Normal_Anomaly 01 June 2011 11:06:17AM *  2 points [-]

If you want to punch babies, then you should not punch babies. (x)

This is WillSawinShould. NormalAnomalyShould says the same thing, because we're both humans. #$%^$_Should, where #$%^$ is the name of an alien from planet Mog, may say something completely different. You and I both use the letter sequence s-h-o-u-l-d to refer to the output of our own unique should-functions.

Lukeprog, the above is how I understand your post. Is it correct?

Comment author: Will_Sawin 01 June 2011 11:21:58AM *  2 points [-]

No. We both use the letter sequence "should" to direct our actions.

We believe that we should follow the results of our should-functions. We believe that the alien from Mog is wrong to follow the results of his should-function. These are beliefs, not definitions.

Imagine if you said "The sun will rise tomorrow" and I responded:

"This is NormalAnomaly_Will. WillSawin_Will says the same thing, because we're both humans. #$%^$_Will, where #$%^$ is the name of an alien from planet Mog, may say something completely different. You and I both use the letter sequence w-i-l-l to refer to the output of our own unique will-functions."

Comment author: wedrifid 01 June 2011 12:16:57PM 2 points [-]

Normal_Anomaly's ontology is coherent. What you describe regarding beliefs is also coherent but refers to a different part of reality space than what Normal is trying to describe.

Comment author: Will_Sawin 01 June 2011 03:52:18PM 2 points [-]

I don't understand what "ontology" and "reality space" mean in this context.

Here's a guess:

You're saying that the word "WillSawin_Should" is a reasonable word to use. It is well-defined, and useful in some contexts. But Plain-Old-Should is also a word with a meaning that is useful in some contexts.

in which case I would agree with you.

Comment author: wedrifid 01 June 2011 04:07:48PM 2 points [-]

I was trying to convey that when you speak of beliefs and determination of actions you are describing an entirely different concept than what Normal_Anomaly was describing. To the extent that presenting your statements as a contradiction of Normal's is both a conversational and epistemic error.

Comment author: wedrifid 01 June 2011 11:13:43AM *  1 point [-]

You can write_underscored_names by escaping the _ by preceding it with a \.

Comment author: Will_Sawin 01 June 2011 10:15:10PM 1 point [-]

I'm not really sure why this was downvoted, compared to everything else I've written on the topic.

Did it have to do with the excessive bolding somehow? Do my claims sound especially stupid stated like this?

Comment author: wedrifid 01 June 2011 10:41:30PM 0 points [-]

I'm not really sure why this was downvoted, compared to everything else I've written on the topic.

It seems to completely miss Normal_Anomaly's point, speaking right past him. As to the 'compared to everything else you have written' I refrained from downvoting your replies to myself even though I would have downvoted them if they were replies to a third party. It is a general policy of mine that I find practical, all else being equal.

Comment author: Antisuji 01 June 2011 06:27:04AM *  1 point [-]

So you're defining "should" to describe actions that best further one's terminal values? Or is there an additional "shouldness" about terminal values too?

Also, regarding

Because if "ought" determines our [proper] behavior, and we can define what "ought" means, then we can define proper behavior into existence

in the grandparent, it sounds like you're equivocating between defining what the word "ought" means and changing the true nature of the concept that "ought" usually refers to. (Unless I was wrong to add the "proper" in the quote, in which case I actually don't know what point you were making.) To wit: "ought" is just a word that we can define as we like, but the concept that "ought" usually refers to is an adaptation and declaring that "ought" actually means something different will not change our actual behavior, except insofar as you succeed in changing others' terminal values.


Incidentally this is a very slippery topic for me to talk about for reasons that I don't fully understand, but I suspect it has to do with my moral intuitions constantly intervening and loudly claiming "no, it should be this way!" and the like. I also strongly suspect that this difficulty is nearly universal among humans.

Comment author: Will_Sawin 01 June 2011 11:04:16AM 1 point [-]

Or is there an additional "shouldness" about terminal values too?

There is.

(Unless I was wrong to add the "proper" in the quote, in which case I actually don't know what point you were making.)

You weren't.

in the grandparent, it sounds like you're equivocating between defining what the word "ought" means and changing the true nature of the concept that "ought" usually refers to.

I do not think I am equivocating. Rather, I disagree with lukeprog about what people are changing when they disagree about morality.

lukeprog thinks that people disagree about what ought means / the definition of ought.

I believe that (almost) everybody things "ought" means the same thing, and that people disagree about the concept that "ought" usually refers to.

This concept is special because it has a reverse definition. Normally a word is defined by the situations in which you can infer that a statement about that word is true. However, "ought" is defined the other way - by what you can do when you infer that a statement about "ought" is true.

Is it the case that Katy ought to buy a car? Well, I don't know. But I know that if Katy is rational, and she becomes convinced that she ought to buy a car, then she will buy a car.

Comment author: prase 01 June 2011 03:50:09PM *  0 points [-]

I believe that (almost) everybody things "ought" means the same thing, and that people disagree about the concept that "ought" usually refers to.

What is the difference between what "ought" means and what it refers to?

Edit:

This concept is special because it has a reverse definition. Normally a word is defined by the situations in which you can infer that a statement about that word is true. However, "ought" is defined the other way - by what you can do when you infer that a statement about "ought" is true.

In the above, do you say that "You ought to do X." is exactly equivalent to the command"Do X!", and "I ought to do X." means "I will do X on the first opportunity and not by accident." ?

Is it the case that Katy ought to buy a car? Well, I don't know. But I know that if Katy is rational, and she becomes convinced that she ought to buy a car, then she will buy a car.

Ought we base the definition of "ought" on a pretty complicated notion of rationality?

Comment author: Will_Sawin 01 June 2011 03:55:06PM 1 point [-]

(yay, I finally caused a confusion that should be really easy to clear up!)

Alice and Bob agree that "Earth" means "that giant thing under us". Alice and Bob disagree about the Earth, though. They disagree about that giant thing under them. Alice thinks it's round, and Bob thinks it's flat.

Comment author: Antisuji 01 June 2011 06:12:42PM 1 point [-]

Yes, this is the distinction I had in mind.

Comment author: prase 01 June 2011 04:02:19PM 0 points [-]

I have difficulty to apply the analogy to ought.

Comment author: Will_Sawin 01 June 2011 04:07:52PM 1 point [-]

In the above, do you say that "You ought to do X." is exactly equivalent to the command"Do X!", and "I ought to do X." means "I will do X on the first opportunity and not by accident." ?

To the first one, yes, but they have different connotations.

To the second one, sort of. "I" can get fuzzy here. I have akrasia problems. I should do my work, but I will not do it for a while. If you cut out a sufficiently small portion of my mind then this portion doesn't have the opportunity to do my work until it actually does my work, because the rest of my mind is preventing it.

Furthermore I am thinking about them more internally. "should" isn't part of predicting actions, its part of choosing them.

Ought we base the definition of "ought" on a pretty complicated notion of rationality?

It doesn't seem complicated to me. Certainly simpler than lukeprog's definitions.

These issues are ones that should be cleared up by the discussion post I'm going to write in a second.

Comment author: prase 01 June 2011 04:13:40PM 0 points [-]

These issues are ones that should be cleared up by the discussion post I'm going to write in a second.

It seems that my further questions rather ought to wait a second, then.

Comment author: Will_Sawin 01 June 2011 06:03:45PM 1 point [-]

I don't want to spam but if people haven't noticed then hopefully this comment should inform them that my first-ever lesswrong post, which might or might not make this clearer, is up.

Comment author: wedrifid 01 June 2011 12:25:13PM *  1 point [-]

We must give up one, and I say give up yours.

I would much prefer to keep Luke's. Basically because it is is actually useful when communicating with others who aren't interested in having the other person's values rammed down their throat. Because if you went around saying an ought at me using your definition then obviously you should expect me to reject it regardless of what the content is. Because the way you are using the term is such that it assumes that the recipient is ultimately subject to something that refers to your own mind.

Comment author: Will_Sawin 01 June 2011 04:00:32PM 1 point [-]

So if you tell me I should go do something, and I agree with you, and I never go do that, you would see nothing inconsistent?

I'm totally comfortable with claims of the form "If you believe XYZ normative statements, then you should do W." It should work just as well as conditionals about physical statements.

Comment author: wedrifid 01 June 2011 04:10:46PM *  0 points [-]

So if you tell me I should go do something, and I agree with you, and I never go do that, you would see nothing inconsistent?

No, that is not something that is implied by my statements.

It is an example of someone not acting according to their own professed ideals and is inconsistent in the same way that all such things are.

Comment author: Will_Sawin 01 June 2011 04:20:01PM 1 point [-]

So you're saying that I am only allowed to use "should" to mean "WillSawin_should". I can't use it to mean "wedrifid_should".

This seems like an odd way to run a conversation to me.

Comment author: wedrifid 01 June 2011 04:29:54PM *  0 points [-]

So you're saying that I am only allowed to use "should" to mean "WillSawinshould". I can't use it to mean "wedrifidshould".

No, that is another rather bizarre thing which I definitely did not say. Perhaps it will be best for me to just leave it with my initial affirmation of Luke's post:

We must give up one, and I say give up yours.

I would much prefer to keep Luke's.

In my observation Luke's system for reducing moral claims provides more potential for enabling effective communication between agents and a more comprehensive way to form a useful epistemic model of such conversations.

Comment author: Will_Sawin 01 June 2011 05:10:30PM 1 point [-]

No, that is another rather bizarre thing which I definitely did not say. Perhaps it will be best for me to just leave it with my initial affirmation of Luke's post:

So suppose I say:

"I wedrifid_should do X" and then don't do X. Clearly, I am not being inconsistent.

but if I say:

"I should do X" and then don't do X then I am being inconsistent.

Something must therefore prevent me from using "should" to mean "wedrifid_should".

Comment author: Manfred 01 June 2011 10:34:46PM *  0 points [-]

I'd agree that you can (and probably do) use plain old "should" to mean multiple things. The trouble is that this isn't very useful for communication. So when communicating, us humans use heuristics to figure out what "should" is meant.

In the example of the conversation, if I say "you should X" and you say "I agree," then I generally use a shortcut to think you meant Will-should. The obvious reason for this is that if you meant Manfred-should, you would have just repeated my own statement back to me, which would be not communicating anything, and it's a decent shortcut to assume that when people say something they want to communicate. The only other obvious "should" in the conversation is Will-should, so it's a good guess that you meant Will-should.

Comment author: Will_Sawin 01 June 2011 11:31:10PM 2 points [-]

"I agree" generally means the same thing as repeating someone's statement back at them. We can expand:

"You wedrifd_should do X"

"I agree, I will_should do X"

I seem to be making an error of interpretation here if I say things the way you normally say them! Why, in this instance, is it considered normal and acceptable to interpret professed agreement as expressing a different belief than the one being agreed to?

It all seems fishy to me.

Comment author: Manfred 02 June 2011 12:14:25AM *  0 points [-]

Huh, yeah that is weird. But on thinking about it, I can only think of two situations I've heard or used "I agree." One is if there's a problem with an unsure solution, where it means "My solution-finding algorithm also returned that," and if someone offers a suggestion about what should be done, where I seem to be claiming it usually means "My should-finding algorithm also returned that."

Comment author: Peterdjones 01 June 2011 06:48:06PM 0 points [-]

But my dialogs and your dialogs contradict each other! Because if "ought" determines our behavior, and we can define what "ought" means, then we can define proper behavior into existence

Moral ideas don't determinne behaviour with any great reliability, so there is no analytical or necessary relationship there. If that's what you were getting at.

Comment author: Perplexed 01 June 2011 02:05:03AM *  4 points [-]

In The Is_Ought Gap, Luke writes

If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not. If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest.

Ironically, this is where I quickly lost interest in this article, because glib word-play isn't my subject of interest.

Comment author: lukeprog 08 June 2011 07:00:58AM 1 point [-]

I wasn't trying to be glib. All I'm saying is that if somebody uses 'ought' terms to refer to things that can't be reduced to physics, then they're either talking about things that don't exist or they aren't physicalists - and physicalism vs. non-physicalism is beyond the scope of this particular article.

Comment author: Perplexed 08 June 2011 02:36:45PM 0 points [-]

... if somebody uses 'ought' terms to refer to things that can't be reduced to physics ...

Your emphasis on can't. But do you really mean "can't be reduced"? Or are you rather excluding anything that "hasn't been reduced"? In the original posting you seem to be demanding that anyone using "ought language" be prepared to perform the reduction on the spot:

If they are making a claim about the world of is, then I ask them which part of the world of is they are discussing. I ask which ought-reductionism they have in mind.

I'm sorry, Luke. My commitment to reductionism is more of a vague hope for future physicalist explanation. I did not agree to use only that language which I can simultaneously show to be universally tabooable.

Comment author: lukeprog 08 June 2011 05:27:31PM 2 points [-]

Perplexed,

I feel you're being uncharitable. Do you really think we're having a disagreement about how reductionism works? I'm not demanding that somebody have a full down-to-atoms reduction in mind whenever they use any term whatever. My target in that paragraph is people who themselves don't think their use of 'ought' will ever reduce into physics - for example Richard Chappell.

Comment author: Perplexed 08 June 2011 05:40:53PM *  1 point [-]

I feel you're being uncharitable.

I believe that you do feel that. But if you think that excluding people like Chappell from the discussion is a fruitful way to proceed, then I am curious why you believe that what you are discussing is properly termed "Meta-ethics".

ETA: Ok, I'll try for a bit more charity. Why does it matter whether a reduction of "ought" to physics can or cannot be accomplished? Why not simply present your ideas and then point out that they have the virtue of making 'ought' reducible?

It seems to me that you are trying too hard to be a good follower of Eliezer. Take my word for it; that isn't necessary to be respected in this forum. It is possible (though admittedly sometimes expensive) to communicate without first achieving some kind of metaphysical solid ground.

Comment author: lukeprog 08 June 2011 10:37:27PM 1 point [-]

I didn't spend more time on reductionism because that is covered in the reductionism sequence. This post is already too long; I don't have the space to re-hash all the arguments for reductionism. Chappell already knows what's in the reductionism sequence, and why he disagrees with it, and that is a different discussion.

Comment author: timtyler 09 June 2011 11:04:45PM *  0 points [-]

Luke's position seems more reasonable here.

I have to say that the "following" around here is kinda irritating to me too.

There's a scene from The Life of Brian that comes to mind - the "you've got to think for yourselves" scene.

Comment author: Jonathan_Graehl 01 June 2011 05:09:22AM 1 point [-]

Can you elaborate?

Comment author: Perplexed 01 June 2011 05:00:09PM 2 points [-]

Can you elaborate?

"World of is" vs "World of is not" is a false dichotomy.

"I lose interest" is possibly the worst and most uncharitable of all forms of philosophical rhetoric. (Hence my 'turnabout'.)

Luke gave a false impression of what non-naturalistic ethics comprises by providing only a single example of such a position - a position which was absurd because it presumed the existence of a deity.

Even Hume allowed that works of mathematics need not be "committed to the flames". Mathematics, to my mind, does not deal with the "world of is", neither does it deal with the "world of is not". Yet if someone were to provide a non-reductionist, but axiomatic, 'definition' of 'good' using the methods and practices of mathematical logic, I certainly would not dismiss it as "uninteresting".

Comment author: Matt_Simpson 01 June 2011 06:30:47PM 1 point [-]

But mathematics does deal with the "world of is" either potentially or as "rules of thought" (all thoughts are in minds). God, on the other hand, is different.

Comment author: RichardWein 27 September 2011 12:31:08AM 1 point [-]

[Re-post with correction]

Hi Luke,

I've questioned your metaethical views before (in your "desirist" days) and I think you're making similar mistakes now as then. But rather than rehash old criticisms I'd like to make a different point.

Since you claim to be taking a scientific or naturalized approach to philosophy I would expect you to offer evidence in support of your position. Yet I see nothing here specifically identified as evidence, and very little that could be construed as evidence. I don't see how your approach here is significantly different from the intuition-based philosophical approaches that you've criticised elsewhere.

Some people who say "Stealing is wrong" are really just trying to express emotions: "Stealing? Yuck!" Others use moral judgments like "Stealing is wrong" to express commands: "Don't steal!" Still others use moral judgments like "Stealing is wrong" to assert factual claims, such as "stealing is against the will of God" or "stealing is a practice that usually adds pain rather than pleasure to the world."

How do you know this? Where's the evidence? I don't doubt that some people say, "Stealing is wrong because it's against the will of God". But where's the evidence that they use "Stealing is wrong" to mean "Stealing is against the will of God"?

But moral terms and value terms are about what we want.

How do you know? And this seems to contradict your claim above that some people use "Stealing is wrong" to mean "stealing is against the will of God". That's not about what we want. (I say that moral terms are primarily about obligations, not wants.)

Comment author: [deleted] 27 September 2011 12:37:21AM *  1 point [-]

You can edit (pencil on paper icon) your posts, you don't have to delete and repost.

Comment author: lukstafi 01 June 2011 11:03:53AM *  1 point [-]

I miss the discussion (on LW in general) of an approach to ethics that strives to determine what actions should be unlawful for an agent, as opposed to, say, what probability distribution over actions is optimal for an agent. (And I don't mean "deontologic", as the "unlawfulness" can be predicated on the consequences.) If you criticize this comment for confusion of "descriptive ethics vs. normative ethics vs. metaethics", try to be constructive.

Comment author: gjm 01 June 2011 03:09:55PM 2 points [-]

Please explain why you think there should be more of that on LW.

Comment author: Peterdjones 01 June 2011 07:11:52PM *  0 points [-]

My answer to that question is that it what morality is actually about, and that personal preference-optimisation is something else.

Comment author: gjm 01 June 2011 09:31:18PM 1 point [-]

I don't think Luke, at least, is conflating morality with personal preference-optimization. He's saying: Different people have different notions of "should"-ness, and if someone says "What should I do?" then giving them a good answer has to begin with working out what notion of "should" they're working with. That applies whether "should" is being used morally or prudentially or both.

Also: What makes a moral agent a moral agent is having personal preferences that give substantial weight to moral considerations. And what such an agent is actually deciding, on any given occasion, is what serves his/her/its goals best: it's just that among the important goals are things like "doing what is right" and "not doing what is wrong". So, actually, for a moral agent "personal preference-optimization" will sometimes involve a great deal of "what morality is actually about".

Comment author: Peterdjones 01 June 2011 09:46:06PM 0 points [-]

There's an important difference between saying preferences may or may not include moral values, and saying morality is, by definition, preference-maximiisation.

Comment author: gjm 01 June 2011 10:00:06PM 0 points [-]

Yup, there is. Did anyone say that morality is, by definition, preference-maximization?

Comment author: Peterdjones 01 June 2011 10:47:33PM 1 point [-]

Yes.

Comment author: gjm 01 June 2011 11:43:58PM 0 points [-]

Do please feel free to provide more information.

Comment author: lukstafi 02 June 2011 12:42:37PM *  0 points [-]

What: discussion of the "social contract" aspect of ethics, for example of the right to not have one's options (sets of actions) constrained beyond a threshold X, what that threshold should be, e.g. a property that the actions that infringe right-X of others are forbidden-X.

Why should there be more of that on LW: (1) it is equally practical and important aspect of ethics as the self-help aspect, (1) it seems to be simpler than determining optimal well-being conditions.

Comment author: wedrifid 02 June 2011 03:07:27PM *  5 points [-]

Why should there be more of that on LW: (1) it is equally practical and important aspect of ethics as the self-help aspect

It would call for a series of 'self help' style of posts explaining:

  • The benefits of creating boundaries between your own identity and other people's declarations of wrongness.
  • The art of balancing freedom with political expedience when dealing with other agents who are attempting to coerce you socially.
  • How to maintain internal awareness of the distinction between what you do not do for fear of social consequences vs what you do not do because of your own ethical values.
  • The difference between satisfying the preferences of others vs acquiescing to their demands. Included here would be how to deal with those who haven't developed the ability to express their own desires except indirectly via the declarations of what it is 'right' for others do.
Comment author: MatthewBaker 03 June 2011 10:51:29PM 0 points [-]

On the other hand, when i look at self help i see something i will continue to delay/slowly progress at a constant rate because my current situation seems to be quite similar to my ideal situation. I think that once you reach that point is when you start a more constant but passive process of improvement.

Comment author: gjm 02 June 2011 01:18:44PM 0 points [-]

Your first #1 doesn't seem to me to be a good justification for having more of it on LW. Lots of things are practical and important but don't belong on LW.

Your second #1 seems to me wrong; deciding what's actually right and wrong is very much not "simpler than determining optimal well-being conditions", for the following reasons. (a) It's debatable whether it's even meaningful (since many people here are moral nonrealists or relativists of one sort or another). (b) There is no obvious way to reach agreement on what actually influences what's right and what's wrong. Net preference satisfaction? The will of a god? Obeying some set of ethical principles somehow built into the structure of the universe? Or what? (c) Most of the theories held by moral realists about what actually matters make it extraordinarily difficult to determine, in difficult cases, whether a given thing is right or wrong. Utilitarianism requires you to sum (or average, or something) the utilities of perhaps infinitely many beings, over a perhaps infinite extent of time and space. The theory Luke calls "desirism" requires you to work out the consequences of having many agents adopt any possible set of preferences. Intuitionist theories and divine-command theories make the details of what's right and wrong entirely inaccessible. Etc.

Now, perhaps in fact you have some specific meta-ethical theory in mind such that, if that theory is true, then the ethical calculations become manageable. In that case, you might want to say what that meta-ethical theory is and why you think it makes the calculations manageable :-).

Comment author: wedrifid 01 June 2011 11:59:43AM 1 point [-]

I miss the discussion (on LW in general) of an approach to ethics that strives to determine what actions should be unlawful for an agent, as opposed to, say, what probability distribution over actions is optimal for an agent. (And I don't mean "deontologic", as the "unlawfulness" can be predicated on the consequences.) If you criticize this comment for confusion of "descriptive ethics vs. normative ethics vs. metaethics", try to be constructive.

I don't criticize your comment on the basis of any confusion. It appears be more or less a coherent indication of preference. I criticize it based on considering the state which you desire to be both abhorrent and not (sufficiently) lacking here.

Comment author: lukstafi 01 June 2011 02:22:19PM 0 points [-]

Do you find the "classification problem" variant of the "optimization problem" already repugnant, or is it something deeper?

Comment author: wedrifid 01 June 2011 02:39:01PM 0 points [-]

Do you find the "classification problem" variant of the "optimization problem" already repugnant, or is it something deeper?

Classification vs optimization is not necessarily a feature I was commenting on.

Comment author: Peterdjones 07 January 2013 01:23:01AM *  -1 points [-]

But whatever our intended meaning of 'ought' is, the same reasoning applies. Either our intended meaning of 'ought' refers (eventually) to the world of maths and physics (in which case the is-ought gap is bridged), or else it doesn't (in which case it fails to refer).12

The is-ought problem is an epistemic problem. Being informed that some A is ultimately, ontologically, the same as some B does not tell me how that A entails that B. If I cannot see how an "is" implies an "ought", being informed that the "ought" ultimately refer to states of the world -- states of the world far too complex for me to include in my epistemic calculations -- does not help. I can't cram a (representation of a) world-state into my brain. Being informed that if I could I would no longer have an is-ought problem under the unlikely circumstances that I could do so doesn't help. The ontological claim that is's and ought's ultimately have the same referents can only be justified by some epistemic procedure. That is the only way any ontological claim is justified.

Comment author: OrphanWilde 07 January 2013 07:13:43PM 1 point [-]

You really shouldn't be using your own comments as evidence in an argument. It makes your reasoning appear... just a little motivated.

Comment author: Peterdjones 07 January 2013 09:54:58PM -2 points [-]

An argument works or it doens''t.

Comment author: OrphanWilde 07 January 2013 10:03:04PM 1 point [-]

That's true. Which means you really should have brought this argument up and resolved it, instead of making this argument and then declaring the matter unresolved.

Comment author: Peterdjones 07 January 2013 10:18:44PM -2 points [-]

I didn't declared anything unresolved. I have argued that PMR does not close the is-ought gap. AFAIC that stands until someone counterargues. But hey, you could always downvote it out of visibiliy.

Comment author: Ichoran 19 November 2013 11:17:57PM *  1 point [-]

Although I think this series of posts is interesting and mostly very well reasoned, I find the discussion about objectivity to be strangely crafted. At the risk of arguing about definitions: the hierarchy you lay out about objectivity is only remotely related to what I mean by objective, and my sense is that it doesn't cohere very well with common usage.

First, there seems no better reason to split off objective1 than objectiveA which is "software-independent facts". Okay, so I can't say anything objective about my web browser, just because we've said I can't. Why is this helpful? The only reason to split this out is if you are some sort of dualist; otherwise the mind is a computational phenomenon just like DNA replication or whatnot.

Second, as Emile already pointed out, nowhere in the hierarchy is uniqueness addressed, yet this is the clearest conventional distinction between subjectivity and objectivity. 5+7 = 12 for everyone. Mint chocolate chip ice cream is better than rocky road ice cream is not the case for everyone (in the conventional sense, anyway). So these things are all colloquially objective:

  • Rocky road has more chocolate than mint chocolate chip
  • The author of this post enjoys mint chocolate chip more than rocky road
  • My IPv4 address has a higher value than does lesswrong.org
  • The Bible describes God endorsing the consumption of only certain animals

Referring to God doesn't make things non-objective in the standard sense presuming God exists. Of course, without a way to measure God's preferences, you may lose your theoretical objectivity, but any other single source or self-consistent group can fill in (e.g. the Pope) as an source for objective answers to what would otherwise be subjective questions.

The issue isn't whether that is subjective or objective; it's whether that method of gaining objectivity is practical and useful.

And since humans are the only sentient beings, I really fail to see what the distinction is between 2 and 3 is in a practical way, once you split off God (or any other singularly identifiable entity).

So I strongly suggest that this section ought to be rethought. Objectivity seems central to this sort of moral reductionism, and so it is worth using definitions that are not too misleading. Either the definitions should change, or there should be much more motivation about why we care about the distinctions between any of the definitions you've offered.

Comment author: Wei_Dai 15 June 2011 01:26:16AM *  1 point [-]

I'm commenting on the post-change "Is-Ought" section. It seems to me that most of the examples given of "ought" reductions do not support the conclusion that "the is-ought gap can be bridged", because the reductions are wrong. Anyone can propose a naturalistic definition of "ought", but at a minimum, to be right a translation of an "ought" statement into an "is" statement has to preserve the truth value of the "ought" statement, and most of the reductions listed fail to do so.

Take the first example:

"X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z."

If you give me any specific proposal for Y and Z, I'm pretty sure I can find an X such that "you ought to X" is obviously false and "X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z" is true, or vice versa.

Comment author: Garren 02 June 2011 12:22:32AM 1 point [-]

Sounds like a form of speaker relativism, with the 'empathetic' project being about going beyond merely saying that people are expressing different fundamental standards, values, etc. to developing ways to bring those out into the open.

Comment author: TimFreeman 01 June 2011 05:05:33PM 1 point [-]

If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not.

When people are talking about 'ought', they are frequently mean something that's different from 'is' but is like 'is' in that it's a primary concept. For them, 'ought' is not something that can be defined in terms of 'is'.

So IMO people who are talking about 'ought' often really are talking about the world of 'ought', and that's about all you can say about it.

If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest.

You're entitled to be uninterested in the world of 'ought' as a primary concept as well. I am not interested in it either, so I can't defend the point of view of these 'ought' believers. I have repeatedly had conversations with them, so I am sure they exist.

Comment author: Psychosmurf 03 January 2014 07:32:14PM 0 points [-]

The Yudkowskian response is to point out that when cognitivists use the term 'good', their intuitive notion of 'good' is captured by a massive logical function that can't be expressed in simple statements

This is the weakest part of the argument. Why should anybody believe that there is a super complicated function that determines what is 'good'? What are the alternative hypotheses?

I can think of a much simpler hypothesis that explains all of the relevant facts. Our brains come equipped with a simple function that maps "is" statements to "ought" statements. Thus, we can reason about "ought" statements just like we do with "is" statements.

The special thing about this function is that there is nothing special about it at all. It is absolutely trivial. Any "ought" statement can potentially be inferred from any "is" statement. Therefore, "ought" statements can never be conditioned by evidence. This explains not only why there is lots of disagreement among people about what is "good" and that our beliefs about what constitutes "good" will be very complicated, but also that there will be no way to resolve these disagreements.

Comment author: imbatman 01 May 2013 08:15:07PM 0 points [-]

Did the next few posts Luke mentions would be about empathic metaethics ever get written? I don't see them anywhere.

Comment author: lukeprog 01 September 2013 08:09:38PM 1 point [-]

Now that I'm running MIRI, I'll probably never write it, but here's where I was headed.

Comment author: theyangist 15 March 2013 05:03:17PM *  0 points [-]

In jest, I'm going to accuse you of plagiarizing my work, then tell you two problems that I have with the approach that you've outlined, and then wax e-peen and say that mine is similar, but more instructive on moral discourse among all users of it.

My problem here is that we already have a common language (in terms of wants) which reduces "should" and provides the kind of plurality that you're seeking out of this approach, so there's no need to claim, "I'm using 'is good' to mean P," and then eek out a true statement whose truth is a matter of lexical elaboration, when instead people use moral language to alter others' of their own behavior, perspective, etc. without all of that theorizing on top of it. Most people don't make moral arguments on the basis of grand (meta)ethical stances. But it seems like everyone would have to be a deep ethicist to get any traction out of this theory, and that would mean that it really only explains how experts use moral language, but not how everyday people do. But why would one have to be deep about ethics to prescribe that someone do something? Can't people prescribe an action and justify it without appealing to a definition of 'goodness' at all?

My last issue here is a potential paradox that I spotted when I made two pluralistic moral reductionists confront each other:

PMR1: "Is Harris's defined 'being good' better than Craig's defined 'being good'?"

PMR2: "What do you mean by 'better'?"

PMR1: "I mean whatever you mean when you say 'better' in questions like this."

PMR2: "But by 'better', I mean whatever you mean when you say 'better' in questions like this."

Comment author: luke_turner 24 April 2012 03:56:09PM 0 points [-]

I'm confused. If this article promotes pluralistic moral reduction, why does Luke M. make a statement that sounds reductionaly singular rather than plural? I mean this statement here:

But moral terms and value terms are about what we want.

Isn't this a reduction of morality to a single thing, specifically, "what we want", i.e., desire?

Isn't this defining morality as the practice of acting on our desires? And does not this definition contradict other definitions/reductions?

Comment author: syzygy 08 March 2012 07:37:01AM 0 points [-]

Am I correct in (roughly) summarizing your conclusion in the following quote?

Yes, there really is morality, and we can locate it in reality — either as a set of facts about the well-being of conscious creatures, or as a set of facts about what an ideally rational and perfectly informed agent would prefer, or as some other set of natural facts.

If so, what is the logical difference between your theory and moral relativism? What if a person's set of natural facts for morality is "those acts which the culture I was born into deem to be moral"?

Comment author: lukeprog 24 June 2011 07:53:55PM 0 points [-]

Update: I've had the pleasure of discussing some of the topics from this post in episode 87 of video-podcast 'Truth-Driven Thinking.'

Comment author: smijer 06 June 2011 02:28:22AM *  0 points [-]

Stealing often means wrongful taking of property... but point well taken.

Comment author: lukeprog 08 June 2011 07:10:33AM 0 points [-]

True! Maybe I need a different example.

Comment author: lukeprog 14 June 2011 05:37:34PM *  0 points [-]

I'm about to rewrite the section on the is-ought gap, for clarity, so here's a copy of the original text:

Many claim that you cannot infer an 'ought' statement from a series of 'is' statements. The objection comes from Hume, who wrote that he was surprised whenever an argument made of is and is not propositions suddenly shifted to an ought or ought not claim without explanation.

Many of Hume's followers concluded that the problem is not just that this shift happens without adequate explanation, but that one can never derive an 'ought' claim from a series of 'is' claims.

But why should this be? If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not. (What they're talking about is plausibly reducible to physics, or else it isn't.) If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest. If they are making a claim about the world of is, then I ask them which part of the world of is they are discussing. I ask which ought-reductionism they have in mind.

Often, they have in mind a common ought-reductionism known as the hypothetical imperative. This is an ought of the kind: "If you desire to lose weight, then you ought to consume fewer calories than your burn." (But usually, people leave off the implied if statement, and simply say "You should eat less and exercise more.")

A hypothetical imperative (a kind of ought statement) reduces in a straightforward way to a prediction about reality (a kind of is statement). "If you desire to lose weight, then you ought to consume fewer calories than you burn" translates to the claim "If you consume fewer calories than you burn, then you will (or are, ceteris paribus, more likely to) fulfill your desire to lose weight."

Or, perhaps someone has a moral reductionism in mind during a particular use of 'ought' language. Perhaps by "You ought to be more forgiving" they really mean "If you are more forgiving, this is likely to increase the amount of pleasure in the world."

As you can see, it is not hard to bridge the is-ought gap. 'Ought' statements either collapse into the world of is (as with hypothetical imperatives or successful moral reductionisms), or else they collapse into the world of is not (as with Craig's moral theory of divine approval).

Comment deleted 05 June 2011 06:51:11PM *  [-]
Comment author: Wei_Dai 05 June 2011 09:45:06PM *  2 points [-]

If I am right that the whole purpose of this sequence has to do with friendly AI research,

Luke confirms your guess in the comments section of this post.

then how is it useful to devote so much resources to explaining the basics, instead of trying to figure out how to design, or define mathematically, something that could extrapolate human volition?

I can see a couple of benefits of going over the basics first. One, it lets Luke confirm that his understanding of the basics are correct, and two, it can interest others to work in the same area (or if they are already interested, can help bring them up to the same level as Luke). I would like to see Luke continue this sequence. (Unless of course he already has some good new ideas about FAI, in which case write those down first!)

Comment author: XiXiDu 06 June 2011 08:26:59AM *  1 point [-]

But how does ethics matter for friendly AI? If a friendly AI is going to figure out what humans desire, by extrapolating their volition, might it conclude that our volition is immoral and therefore undesirable?

Comment author: Will_Sawin 06 June 2011 11:09:46AM 0 points [-]

What morality unit would it have other than human's volition?

If it has another, separate volition unit, yes.

If not, then only if humans fundamentally self-contradict, which seems unlikely, because biological systems are pretty robust to that.

Comment author: XiXiDu 06 June 2011 11:35:18AM 0 points [-]

What morality unit would it have other than human's volition?

I am not sure what a 'morality unit' is supposed to be or how it would be different from a volition unit. Either morality is part of our volition, instrumental or an imperative. In each case one could ask what we want and arrive at morality.

Comment author: Will_Sawin 06 June 2011 03:51:09PM 3 points [-]

What I'm saying is that: If Clippy tried to calculate our volition, he would conclude that our volition is immoral. (Probably. Maybe our volition IS paperclips.)

But if we programmed an AI to calculate our volition and use that as its volition, and our morality as its morality, and so on, then it would not find our volition immoral unless we find our volition immoral, which seems unlikely.

Comment author: Peterdjones 06 June 2011 04:04:55PM 0 points [-]

An AI that was smarter than us might deduce that we were not applying the Deep Structure of our morality properly because of bias or limited intelligence. It might conclude that human morality requires humans to greatly reduce their numbers in order to lessen the impact on other species, for instance.

Comment author: lukeprog 08 June 2011 07:13:05AM 0 points [-]

I can see a couple of benefits of going over the basics first. One, it lets Luke confirm that his understanding of the basics are correct, and two, it can interest others to work in the same area (or if they are already interested, can help bring them up to the same level as Luke).

Correct!