Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

RichardChappell comments on What is Eliezer Yudkowsky's meta-ethical theory? - Less Wrong Discussion

33 Post author: lukeprog 29 January 2011 07:58PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (369)

You are viewing a single comment's thread.

Comment author: RichardChappell 30 January 2011 08:40:28PM *  40 points [-]

Eliezer's metaethics might be clarified in terms of the distinctions between sense, reference, and reference-fixing descriptions. I take it that Eliezer wants to use 'right' as a rigid designator to denote some particular set of terminal values, but this reference fact is fixed by means of a seemingly 'relative' procedure (namely, whatever terminal values the speaker happens to hold, on some appropriate [if somewhat mysterious] idealization). Confusions arise when people mistakenly read this metasemantic subjectivism into the first-order semantics or meaning of 'right'.

In summary:

(i) 'Right' means, roughly, 'promotes external goods X, Y and Z'

(ii) claim i above is true because I desire X, Y, and Z.

Note that Speakers Use Their Actual Language, so murder would still be wrong even if I had the desires of a serial killer. But if I had those violent terminal values, I would speak a slightly different language than I do right now, so that when KillerRichard asserts "Murder is right!" what he says is true. We don't really disagree, but are instead merely talking past each other.

Virtues of the theory:

(a) By rigidifying on our actual, current desires (or idealizations thereupon), it avoids Inducing Desire Satisfactions.

(b) Shifting the subjectivity out to the metasemantic level leaves us with a first-order semantic proposal that at least does a better job than simple subjectivism at 'saving the phenomena'. (It has echoes of Mark Schroeder's desire-based view of reasons, according to which the facts that give us reasons are the propositional contents of our desires, rather than the desires themselves. Or something like that.)

(c) It's naturalistic, if you find moral non-naturalism 'spooky'. (Though I'd sooner recommend Mackie-style error theory for naturalists, since I don't think (b) above is enough to save the phenomena.)

Objections

(1) It's incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.

(2) The semantic tricks merely shift the lump under the rug, they don't get rid of it. Standard worries about relativism re-emerge, e.g. an agent can know a priori that their own fundamental values are right, given how the meaning of the word 'right' is determined. This kind of (even merely 'fundamental') infallibility seems implausible.

(3) Just as simple subjectivism is an implausible theory of what 'right' means, so Eliezer's meta-semantic subjectivism is an implausible theory of why 'right' means promoting external goods X, Y, Z. An adequately objective metaethics shouldn't even give preferences a reference-fixing role.

Comment author: komponisto 31 January 2011 04:57:59AM *  19 points [-]

I think this is an excellent summary. I would make the following comments:

Confusions arise when people mistakenly read this metasemantic subjectivism into the first-order semantics or meaning of 'right'.

Yes, but I think Eliezer was mistaken in identifying this kind of confusion as the fundamental source of the objections to his theory (as in the Löb's theorem discussion). Sophisticated readers of LW (or OB, at the time) are surely capable of distinguishing between logical levels. At least, I am -- but nevertheless, I still didn't feel that his theory was adequately "non-relativist" to satisfy the kinds of people who worry about "relativism". What I had in mind, in other words, was your objections (2) and (3).

The answer to those objections, by the way, is that an "adequately objective" metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information. This directly answers (3), anyway; as for (2), "fallibility" is rescued (on the object level) by means of imperfect introspective knowledge: an agent could be mistaken about what its own terminal values are.

Comment author: Matt_Simpson 31 January 2011 06:43:14PM *  3 points [-]

Note that your answer to (2) also answers (1): value uncertainty makes it seem as if there is substantive, fundamental normative disagreement even if there isn't. (Or maybe there is if you don't buy that particular element of EY's theory)

Comment author: Yosarian2 30 August 2017 09:44:22AM 0 points [-]

The answer to those objections, by the way, is that an "adequately objective" metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information.

Elizer attempted to deal with that problem by defining a certain set of things as "h-right", that is, morally right from the frame of reference of the human mind. He made clear that alien entities probably would not care about what is h-right, but that humans do, and that's good enough.

Comment author: RichardChappell 01 February 2011 04:56:46AM 0 points [-]

The answer to those objections, by the way, is that an "adequately objective" metaethics is impossible

That's not a reason to prefer EY's theory to an error theory (according to which properly normative properties would have to be irreducibly normative, but no such properties actually exist).

Comment author: lukeprog 01 February 2011 12:51:00PM 3 points [-]

Richard,

Until persuaded otherwise, I agree with you on this point. (These days, I take Richard Joyce to have the clearest defense of error theory, and I just subtract his confusing-to-me defense of fictionalism.) Besides, I think there are better ways of getting something like an 'objective' ethical theory (in something like a 'realist' sense) while still holding that reasons for action arise only from desires, or from relations between desires and states of affairs. In fact, that's the kind of theory I defend: desirism. Though, I'm not too interested anymore in whether desirism is to be called 'objective' or 'realist', even though I think a good case can be made for both.

Comment author: lukeprog 01 February 2011 01:02:07PM 13 points [-]

Richard,

You're speaking my language, thanks! I hope this is EY's view, because I know what this means. Maybe now I can go back and read EY's sequence in light of this interpretation and it will make more sense to me now.

EY's theory as presented above makes me suspicious that making basic evaluative moral terms rigid designators is a kind of 'trick' which, though perhaps not intended, very easily has the effect of carrying along some common absolutist connotations of those terms where they no longer apply in EY's use of those terms.

At the moment, I'm not so worried about objection (1), but objections (2) and (3) are close to what bother me about EY's theory, especially if this is foundational for EY's thinking about how we ought to be designing a Friendly AI. If we're working on a project as important as Friendly AI, it becomes an urgent problem to get our meta-ethics right, and I'm not sure Eliezer has done it yet. Which is why we need more minds working on this problem. I hope to be one of those minds, even if my current meta-ethics turns out to be wrong (I've held my current meta-ethics for under 2 years, anyway, and it has shifted slightly since adoption).

But, at the moment it remains plausible to me that Eliezer is right, and I just don't see why right now. Eliezer is a very smart guy who has invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief.

Comment author: RichardChappell 01 February 2011 05:12:32PM 11 points [-]

invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief

Maybe; I can't say I've noticed that so much myself -- e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven't seen him retract his claim that zombies are demonstrably incoherent.

Comment author: Vladimir_Nesov 01 February 2011 08:12:32PM *  6 points [-]

e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven't seen him retract his claim that zombies are demonstrably incoherent.

Clearly, from his standpoint a lot of things you believed were confused, and he decided against continuing to argue. This is a statement about willingness to engage situations where someone's wrong on the Internet and presence of disagreement, not external evidence about correctness (distinct from your own estimate of correctness of your opponent's position).

Comment author: lukeprog 01 February 2011 10:49:38PM 4 points [-]

You think that "clearly" Eliezer believed many of Richard's beliefs were confused. Which beliefs, do you think?

Comment author: Vladimir_Nesov 01 February 2011 11:50:57PM 11 points [-]

I won't actually argue, just list some things that seem to be points where Richard talks past the intended meaning of the posts (irrespective of technical accuracy of the statements in themselves, if their meaning intended by Richard was what the posts referred to). Link to the post for convenience.

  • "premise that words refer to whatever generally causes us to utter them": There is a particular sense of "refer" in which we can trace the causal history of words being uttered.
  • "It's worth highlighting that this premise can't be right, for we can talk about things that do not causally affect us. ": Yes, we can consider other senses of "refer", make the discussion less precise, but those are not the senses used.
  • "We know perfectly well what we mean by the term 'phenomenal consciousness'.": Far from "perfectly well".
  • "We most certainly do not just mean 'whatever fills the role of causing me to make such-and-such utterances'" Maybe we don't reason so, but it's one tool to see what we actually mean, even if it explores this meaning in a different sense from what's informally used (as a way of dissolving a potentially wrong question).
  • "No, the example of unicorns is merely to show that we can talk about non-causally related things.": We can think/talk about ideas that cause us to think/talk about them in certain ways, and in this way the meaning of the idea (as set of properties which our minds see in it) causally influences uttering of words about it. Whether what the idea refers to causally influences us in other ways is irrelevant. On the other hand, if it's claimed that the idea talks about the world (and is not an abstract logical fact unrelated to the world), there must be a pattern (event) of past observations that causes the idea to be evaluated as "correct", and alternative observations that cause it to be evaluated as "wrong" (or a quantitative version of that). If that's not possible, then it can't be about our world.
Comment author: XiXiDu 01 February 2011 06:50:59PM 1 point [-]

This is the first time I saw anyone telling EY that what he wrote is plainly false.

Comment author: timtyler 02 February 2011 12:41:04AM 2 points [-]
Comment author: Kaj_Sotala 02 February 2011 12:04:52PM 8 points [-]

I agree with the first one of those being bad.

Yes, if you're talking about corporations, you cannot use exactly the same math than you do if you're talking about evolutionary biology. But there are still some similarities that make it useful to know things about how selection works in evolutionary biology. Eliezer seems to be saying that if you want to call something "evolution", then it has to meet these strictly-chosen criteria that he'll tell you. But pretty much the only justification he offers is "if it doesn't meet these criteria, then Price's equation doesn't apply", and I don't see why "evolution" would need to be strictly defined as "those processes which behave in a way specified by Price's equation". It can still be a useful analogy.

The rest are fine in my eyes, though the argument in The Psychological Unity of Humankind seems rather overstated for several reasons.

Comment author: timtyler 02 February 2011 02:50:08PM 2 points [-]

FWIW, cultural evolution is not an analogy. Culture literally evolves - via differential reproductive success of memes...

Comment author: Will_Newsome 15 May 2011 08:34:36AM 6 points [-]

Do you have recommendations for people/books that take this perspective seriously and then go on to explore interesting things with it? I haven't seen anyone include the memetic perspective as part of their everyday worldview besides some folk at SIAI and yourself, which I find pretty sad.

Also, I get the impression you have off-kilter-compared-to-LW views on evolutionary biology, though I don't remember any concrete examples. Do you have links to somewhere where I could learn more about what phenomena/perspectives you think aren't emphasized or what not?

Comment author: timtyler 15 May 2011 08:46:20AM *  9 points [-]

My current project is a book on memetics. I also have a blog on memetics.

Probably the best existing book on the topic is The Meme Machine by Susan Blackmore.

I also maintain some memetics links, some memetics references, a memetics glossary - and I have a bunch of memetics videos.

In academia, memetics is typically called "cultural evolution". Probably the best book on that is "Not by Genes Alone".

Your "evolutionary biology" question is rather vague. The nearest thing that springs to mind is this. Common views on that topic around here are more along the lines expressed in the The Robot's Rebellion. If I am in a good mood, I describe such views as "lacking family values" - and if I am not, they get likened to a "culture of death".

Comment author: Will_Newsome 15 May 2011 09:04:20AM 3 points [-]

Wow, thanks! Glad I asked. I will start a tab explosion.

Comment author: lukeprog 01 February 2011 08:44:53PM 1 point [-]

Really? That's kind of scary...

Comment author: Desrtopa 01 February 2011 08:49:08PM *  1 point [-]

His response to it, or that it's done so infrequently?

I for one am less worried the less often he writes things that are plainly false, so his being called out rarely doesn't strike me as a cause for concern.

Comment author: lukeprog 01 February 2011 09:28:46PM 11 points [-]

What scares me is that people say EY's position is "plainly false" so rarely. Even if EY is almost always right, you would still expect a huge number of people to say that his positions are plainly false, especially when talking about such difficult and debated questions as those of philosophy and predicting the future.

Comment author: wedrifid 03 February 2011 08:05:19AM 15 points [-]

What scares me is that people say EY's position is "plainly false" so rarely.

What scares me is how often people express this concern relative to how often people actually agree with EY. Eliezer's beliefs and assertions take an absolute hammering. I agree with him fairly often - no surprise, he is intelligent, has a similar cognitive style mine and has spent a whole lot of time thinking. But I disagree with him vocally whenever he seems wrong. I am far from the only person who does so.

Comment author: Desrtopa 01 February 2011 09:34:50PM 9 points [-]

If the topics are genuinely difficult, I don't think it's likely that many people who understand them would argue that Eliezer's points are plainly false. Occasionally people drop in to argue such who clearly don't have a very good understanding of rationality or the subject material. People do disagree with Eliezer for more substantive reasons with some frequency, but I don't find the fact that they rarely pronounce him to be obviously wrong particularly worrying.

Comment author: Kaj_Sotala 02 February 2011 11:49:58AM 6 points [-]

Most of the people who are most likely to think that EY's positions on things are plainly false probably don't bother registering here to say so.

There's one IRC channel populated with smart CS / math majors, where I drop LW links every now and then. Pretty frequently they're met with a rather critical reception, but while those people are happy to tear them apart on IRC, they have little reason to bother to come to LW and explain in detail why they disagree.

(Of the things they disagree on, I mainly recall that they consider Eliezer's treatment of frequentism / Bayesianism as something of a strawman and that there's no particular reason to paint them as two drastically differing camps when real statisticians are happy with using methods drawn from both.)

Comment author: lessdazed 29 March 2011 05:03:10AM *  4 points [-]

they consider Eliezer's treatment of frequentism / Bayesianism as something of a strawman and that there's no particular reason to paint them as two drastically differing camps when real statisticians are happy with using methods drawn from both.

In that case, we got very different impressions about how Eliezer described the two camps; here is what I heard: <channel righteous fury of Eliezer's pure Bayesian soul>

It's not Bayesian users on the one hand and Frequentists on the other, each despising the others' methods. Rather, it's the small group of epistemic statisticians and a large majority of instrumentalist ones.

The epistemics are the small band of AI researchers using statistical models to represent probability so as to design intelligence, learning, and autonomy. The idea is that ideal models are provably Baysian, and the task undertaken is to understand and implement close approximations of them.

The instrumentalist mainstream doesn't always claim that it's representing probability and doesn't feel lost without that kind of philosophical underpinning. Instrumentalists hound whatever problem is at hand with all statistical models and variables that they can muster to get the curve or isolated variable etc. they're looking for and think is best. The most important part of instrumentalist models is the statistician him or herself, which does the Bayesian updating adequately and without the need for understanding. </channel righteous fury of Eliezer's pure Bayesian soul>

Saying that the division is a straw man because most statisticians use all methods misses the point.

Edit: see for example here and here.

Comment author: lukeprog 02 February 2011 04:29:33PM 4 points [-]

Most of the people who are most likely to think that EY's positions on things are plainly false probably don't bother registering here to say so.

True, but I still wouldn't expect sharp disagreement with Eliezer to be so rare. One contributing factor may be that Eliezer at least appears to be so confident in so many of his positions, and does not put many words of uncertainty into his writing about theoretical issues.

Comment author: TheOtherDave 02 February 2011 05:06:38PM 11 points [-]

When I first found this site, I read through all the OB posts chronologically, rather than reading the Sequences as sequences. So I got to see the history of several commenters, many of whom disagreed sharply with EY, with their disagreement evolving over several posts.

They tend to wander off after a while. Which is not surprising, as there is very little reward for it.

So I guess I'd ask this a different way: if you were an ethical philosopher whose positions disagreed with EY, what in this community would encourage you to post (or comment) about your disagreements?

Comment author: orthonormal 01 February 2011 05:18:59PM 5 points [-]

It seems to me that EY himself addressed all three of the objections you list (though of course this doesn't imply he addressed them adequately).

(1) It's incompatible with the datum that substantive, fundamental normative disagreement is in fact possible. People may share the concept of a normative reason, even if they fundamentally disagree about which features of actions are the ones that give us reasons.

Moral Error and Moral Disagreement confronts this.

My own thinking is that humans tend to have the same underlying (evolved) structures behind our hard-to-articulate meta-ethical heuristics, even when we disagree broadly on object-level ethical issues (and of course hand-pick our articulations of the meta-criteria to support our object-level beliefs- the whole machinery of bias applies here).

This implies both that my object-level beliefs can be at odds with their meta-level criteria (if this becomes too obvious for me to rationalize away, I'm more likely to change one or other object-level belief than to change the meta-level heuristic), and that you and I can disagree fundamentally on the object level while still believing that there's something in common which makes argumentation relevant to our disagreement.

Comment author: RichardChappell 01 February 2011 05:24:28PM 8 points [-]

Moral Error and Moral Disagreement confronts this

Yeah, I'm the "Richard4" in the comments thread there :-)

Comment author: orthonormal 01 February 2011 06:08:49PM *  11 points [-]

OK. I'll reply here because if I reply there, you won't get the notifications.

The crux of your argument, it seems to me, is the following intuition:

Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents.

This is certainly a property we would want morality to have, and one which human beings naturally assume it must have– but is that the central property of it? Should it turn out that nothing which looks like morality has this property, does it logically follow that all morality is dead, or is that reaction just a human impulse?

(I will note, with all the usual caveats, that believing one's moral sentiments to be universal in scope and not based on preference is a big advantage in object-level moral arguments, and that we happen to be descended from the winners of arguments about tribal politics and morality.)

If a certain set of moral impulses involves shared standards common to, say, every sane human being, then moral arguments would still work among those human beings, in exactly the way you would want them to work across all intelligent beings. Frankly, that's good enough for me. Why give baby-eating aliens in another universe veto powers over every moral intuition of yours?

Comment author: RichardChappell 01 February 2011 06:35:50PM *  5 points [-]

Thanks for the reply -- I find this a very interesting topic. One thing I should clarify is that my view doesn't entail giving aliens "veto powers", as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn't an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)

In any case, I'd put the crux of my argument slightly differently. The core intuition is just that it's possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY's view would seem to imply that the two agents mustn't really disagree. And that just seems a mistake: it's part of our concept of morality that this very concept could be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we're aspiring to conform our judgments to a standard that is outside of ourselves. (If you don't think there are any such objective standards, then that's just to say that there are no normative facts, given my concept of normativity.)

Comment author: cousin_it 01 February 2011 09:14:51PM *  7 points [-]

Richard, hello.

Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program "really exists" somewhere outside a computer, or is "intrinsic" to a computer, or is "contingent", or something like that. Such questions don't correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that's the end of the story.

Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn't imply that your moral intuitions "should" be worthless to you and you "should" seek external authority! There's nothing wrong with mankind living by its internal moral lights.

Yes, it's possible that different computers will have different programs. Our world contains billions of similar "moist robots" running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all "possible" sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that's substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach's conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn't kill babies. Most possible "intelligent agents" (aka algorithms that can hit small targets in large search spaces) aren't humans in funny suits.

Comment author: Vladimir_Nesov 01 February 2011 09:27:30PM *  6 points [-]

I expect Richard's memeset has understanding of all your points that doesn't move his current position. You're probably exposing him to arguments he has already encountered, so there's little point in expecting a different result. I'm not saying that Richard can't be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably "agrees" with a lot of your points, just with a different and more sophisticated understanding than yours.

On the other hand, it might work for the benefit of more naive onlookers.

Comment author: cousin_it 01 February 2011 09:56:38PM *  10 points [-]

The intent of my comment wasn't to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don't.

Comment author: Vladimir_Nesov 01 February 2011 10:00:41PM 2 points [-]

That's a motivation I didn't consider. (Agreed.)

Comment author: RichardChappell 01 February 2011 10:08:49PM *  4 points [-]

Yeah, as Vladimir guessed, this is all familiar.

Your last paragraph suggests that you've misunderstood my view. I'm not making an empirical claim to the effect that all agents will eventually converge to our values -- I agree that that's obviously false. I don't even think that all formally intelligent agents are guaranteed to have normative concepts like 'ought', 'reason', or 'morality'. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other's desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.

(Aside: the further claim about "reasons" in your last sentence presupposes a subjectivist view about reasons that I reject.)

Comment author: cousin_it 01 February 2011 10:32:45PM *  5 points [-]

What use is this concept of "reasonability"? Let's say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it's an "arbitrary omission" and it "should" amend its desires for greater "consistency", the agent just waves you off because listening to you isn't likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn't increase expected utility. If by "rational" you mean something else, I'd like to understand what exactly.

Comment author: RichardChappell 02 February 2011 01:13:52AM *  1 point [-]

I mean 'rational' in the ordinary, indefinable sense, whereby calling a decision 'irrational' expresses a distinctive kind of criticism -- similar to that expressed by the words 'crazy', 'foolish', 'unwise', etc. (By contrast, you can just say "maximizes expected utility" if you really mean nothing more than maximizes expected utility -- but note that that's a merely descriptive concept, not a normative one.)

If you don't possess this concept -- if you never have thoughts about what's rational, over and above just what maximizes expected utility -- then I can't help you.

Comment author: cousin_it 02 February 2011 10:43:37AM *  1 point [-]

I don't think we can make progress with such imprecise thinking. Eliezer has a nice post about that.

Comment author: Vladimir_Nesov 01 February 2011 08:36:29PM *  3 points [-]

The core intuition is just that it's possible to have irresolvable moral disagreements.

What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?

Comment author: orthonormal 01 February 2011 08:38:36PM 4 points [-]

First, EY makes it abundantly clear that two agents can have a fundamental disagreement on values– it's just not the best (or most helpful) assumption when you're talking about two sane human beings with a vast sea of common frameworks and heuristics.

Secondly, I'm worried about what you're trying to do with words when you suggest we "take them to be unreasonable to intrinsically desire the eating of babies".

If you're making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by "unreasonable", then you'd need some strong evidence for that claim.

If instead you openly redefine "reasonable" to include "shares our fundamental moral standards", then the property

it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents

becomes a tautology which no longer excludes "meta-semantic subjectivism", as you put it. So I'm puzzled what you mean.

Comment author: RichardChappell 01 February 2011 10:26:23PM 4 points [-]

Talking past each other a bit here. Let me try again.

EY makes it abundantly clear that two agents can have a fundamental disagreement on values

EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I'm not charging him with being unable to accommodate this. The objection is instead that he's unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.

Your second point rests on a false dichotomy. I'm not making an empirical claim, but nor am I merely defining the word "reasonable". Rather, I'm making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can't make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.

Comment author: Vladimir_Nesov 01 February 2011 08:33:12PM 2 points [-]

an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies

What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don't expect to have object-level understanding of human morality that is substantially more accurate than what you'd get if you add baby-eating impulses to it.

Comment author: orthonormal 01 February 2011 05:42:39PM 1 point [-]

Ah! Sorry for carrying coals to Newcastle, then. Let me catch up in that thread.

Comment author: utilitymonster 31 January 2011 01:39:04PM 5 points [-]

Yes, this is what I thought EY's theory was. EY? Is this your view?

Comment author: Wei_Dai 27 June 2011 05:50:20PM *  4 points [-]

This summary of Eliezer's position seems to ignore the central part about computation. That is, Eliezer does not say that 'Right' means 'promotes external goods X, Y and Z' but rather that it means a specific computation that can be roughly characterized as 'renormalizing intuition'

I see the project of morality as a project of renormalizing intuition. We have intuitions about things that seem desirable or undesirable, intuitions about actions that are right or wrong, intuitions about how to resolve conflicting intuitions, intuitions about how to systematize specific intuitions into general principles.

which eventually outputs something like 'promotes external goods X, Y and Z'. I think Eliezer would argue that at least some of the objections list here are not valid if we add the part about computation. (Specifically, disagreements and fallibility can result from from lack of logical omniscience regarding the output of the 'morality' computation.)

Is the reason for skipping over this part of Eliezer's idea that standard (Montague) semantic theory treats all logically equivalent language as having the same intension? (I believe this is known as "the logical omniscience problem" in linguistics and philosophy of language.)

Comment author: RichardChappell 27 June 2011 08:55:41PM *  3 points [-]

The part about computation doesn't change the fundamental structure of the theory. It's true that it creates more room for superficial disagreement and fallibility (of similar status to disagreements and fallibility regarding the effective means to some shared terminal values), but I see this as an improvement in degree and not in kind. It still doesn't allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.

(I take it to be a metaethical datum that even people with different terminal values, or different Eliezerian "computations", can share the concept of a normative reason, and sincerely disagree about which (if either) of their values/computations is correctly tracking the normative reasons. Similarly, we can coherently doubt whether even our coherently-extrapolated volitions would be on the right track or not.)

Comment author: Wei_Dai 28 June 2011 04:01:05AM 5 points [-]

It still doesn't allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.

It's not clear to me why there must be fundamental disagreement and fallibility, e.g. amongst logically omniscient agents. Can you refer me to an argument or intuition pump that explains why you think that?

Comment author: RichardChappell 30 June 2011 03:55:55PM 3 points [-]

One related argument is the Open Question Argument: for any natural property F that an action might have, be it promotes my terminal values, or is the output of an Eliezerian computation that models my coherent extrapolated volition, or whatever the details might be, it's always coherent to ask: "I agree that this action is F, but is it good?"

But the intuitions that any metaethics worthy of the name must allow for fundamental disagreement and fallibility are perhaps more basic than this. I'd say they're just the criteria that we (at least, many of us) have in mind when insisting that any morality worthy of the name must be "objective", in a certain sense. These two criteria are proposed as capturing that sense of objectivity that we have in mind. (Again, don't you find something bizarrely subjectivist about the idea that we're fundamentally morally infallible -- that we can't even question whether our fundamental values / CEV are really on the right track?)

Comment author: Wei_Dai 02 July 2011 01:57:47AM *  9 points [-]

I'd say they're just the criteria that we (at least, many of us) have in mind when insisting that any morality worthy of the name must be "objective", in a certain sense.

What would you say to someone who does not share your intuition that such "objective" morality likely exists?

My main problem with objective morality is that while it's hard to deny that there seem to be mind-independent moral facts like "pain is morally bad", there doesn't seem to be enough such facts to build an ethical system out of them. What natural phenomena count as pain, exactly? How do we trade off between pain and pleasure? How do we trade off between pain in one person, and annoyance in many others? How do we trade off pain across time (i.e., should we discount future pain, if so how)? Across possible worlds? How do we morally treat identical copies? It seems really hard, perhaps impossible, to answer these questions without using subjective preferences or intuitions that vary from person to person, or worse, just picking arbitrary answers when we don't even have any relevant preferences or intuitions. If it turns out that such subjectivity and/or arbitrariness can't be avoided, that would be hard to square with objective morality actually existing.

(Again, don't you find something bizarrely subjectivist about the idea that we're fundamentally morally infallible -- that we can't even question whether our fundamental values / CEV are really on the right track?)

I do think there's something wrong with saying that we can't question whether CEV is really on the right track. But I wouldn't use the words "bizarrely subjectivist". To me the problem is just that I clearly can and do question whether CEV is really on the right track. Fixing this seems to require retreating quite a bit from Eliezer's metaethical position (but perhaps there is some other solution that I'm not thinking of). At this point I would personally take the following (minimalist) position:

  1. At least some people, at least some of the time, refer to the same concept by "morality" as me and they have substantive disagreements over its nature and content.
  2. I'm not confident about any of its properties.
  3. Running CEV (if it were practical to) seems like a good way to learn more about the nature and content of morality, but there may be (probably are) better ways.
Comment author: Vladimir_Nesov 02 July 2011 10:13:20AM 3 points [-]

If it turns out that such subjectivity and/or arbitrariness can't be avoided, that would be hard to square with objective morality actually existing.

Compare with formal systems giving first-order theories of standard model of natural numbers. You can't specify the whole thing, and at some point you run into (independent of what comes before) statements for which it's hard to decide whether they hold for the standard naturals, and so you could add to the theory either those statements or their negation. Does this break the intuition that there is some intended structure corresponding to natural numbers, or more pragmatically that we can still usefully seek better theories that capture it? For me, it doesn't in any obvious way.

Comment author: Wei_Dai 02 July 2011 04:37:59PM 2 points [-]

It seems to be an argument in favor of arithmetic being objective that almost everyone agree that a certain a set of axioms correctly characterize what natural numbers are (even if incompletely), and from that set of axioms we can derive much (even if not all) of what we want to know about the properties of natural numbers. If arithmetic were in the same situation as morality is today, it would be much harder (i.e., more counterintuitive) to claim that (1) everyone is referring to the same thing by "arithmetic" and "natural numbers" and (2) arithmetic truths are mind-independent.

To put it another way, conditional on objective morality existing, you'd expect the situation to be closer to that of arithmetic. Conditional on it not existing, you'd expect the situation to be closer to what it actually is.

Comment author: RichardChappell 02 July 2011 03:59:52PM 2 points [-]

What would you say to someone who does not share your intuition that such "objective" morality likely exists?

I'd say: be an error theorist! If you don't think objective morality exists, then you don't think that morality exists. That's a perfectly respectable position. You can still agree with me about what it would take for morality to really exist. You just don't think that our world actually has what it takes.

Comment author: Wei_Dai 03 July 2011 12:30:05AM 4 points [-]

Yes, that makes sense, except that my intuition that objective morality does not exist is not particularly strong either. I guess what I was really asking was, do you have any arguments to the effect that objective morality exists?

Comment author: orthonormal 01 February 2011 05:29:21PM *  3 points [-]

(2) The semantic tricks merely shift the lump under the rug, they don't get rid of it. Standard worries about relativism re-emerge, e.g. an agent can know a priori that their own fundamental values are right, given how the meaning of the word 'right' is determined. This kind of (even merely 'fundamental') infallibility seems implausible.

EY bites this bullet in the abstract, but notes that it does not apply to humans. An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of "that-AI-right" in the sense EY is talking about.

But there is no analogue to that situation in human psychology, given how much we now know about self-deception, our conscious and unconscious mental machinery, and the increasing complexity of our values the more we think on them. We can, it's true, say that "the correct extrapolation of my fundamental values is what's right for me to do", but this doesn't guarantee whether value X is or is not a member of that set. The actual work of extrapolating human values (through moral arguments and other methods) still has to be done.

So practical objections to this sort of bullet-biting don't apply to this metaethics; are there any important theoretical objections?

EDIT: Changed "right" to "that-AI-right". Important clarification.

Comment author: TheOtherDave 01 February 2011 06:14:02PM 0 points [-]

Agreed that on EY's view (and my own), human "fundamental values" (1) have not yet been fully articulated/extrapolated; that we can't say with confidence whether X is in that set.

But AFAICT, EY rejects the idea (which you seem here to claim that he endorses?) that an AI with a simple utility function can be sure that maximizing that function is the right thing to do. It might believe that maximizing that function is the right thing to do, but it would be wrong. (2)

AFAICT this is precisely what RichardChappell considers implausible: the idea that unlike the AI, humans can correctly believe that maximizing their utility function is the right thing to do.

==

(1) Supposing there exist any such things, of which I am not convinced.

(2) Necessarily wrong, in fact, since on EY's view as I understand it there's one and only one right set of values, and humans currently implement it, and the set of values humans implement is irreducably complex and therefore cannot be captured by a simple utility function. Therefore, an AI maximizing a simple utility function is necessarily not doing the right thing on EY's view.

Comment author: orthonormal 01 February 2011 08:14:33PM *  0 points [-]

Sorry, I meant to use the two-place version; it wouldn't be what's right; what I meant is that the completely analogous concept of "that-AI-right" would consist simply of that utility function.

Comment author: TheOtherDave 01 February 2011 09:18:40PM 1 point [-]

To the extent that you are still talking about EY's views, I still don't think that's correct... I think he would reject the idea that "that-AI-right" is analogous to right, or that "right" is a 2-place predicate.

That said, given that this question has come up elsethread and I'm apparently in the minority, and given that I don't understand what all this talk of right adds to the discussion in the first place, it becomes increasingly likely that I've just misunderstood something.

In any case, I suspect we all agree that the AI's decisions are motivated by its simple utility function in a manner analogous to how human decisions are motivated by our (far more complex) utility function. What disagreement exists, if any, involves the talk of "right" that I'm happy to discard altogether.

Comment author: Vladimir_Nesov 01 February 2011 08:51:23PM *  0 points [-]

An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of "that-AI-right" in the sense EY is talking about.

I don't think that's right, or EY's position (I'd like evidence on that). Who's to say that maximization is precisely what's right? That might be a very good heuristic, but upon reflection the AI might decide to self-improve in a way that changes this subgoal (of the overall decision problem that includes all the other decision-making parts), by finding considerations that distinguish maximizing attitude to utility and the right attitude to utility. It would of course use its current utility-maximizing algorithm to come to that decision. But the conclusion might be that too much maximization is bad for environment or something. The AI would stop maximizing for the reason it's not the most maximizing thing, the same way as a person would not kill for the reason that action leads to a death, even though avoid-causing-death is not the whole morality and doesn't apply universally.

See also this comment.

Comment author: Kutta 31 January 2011 10:49:06AM *  3 points [-]

(1): I think it's a prominent naturalistic feature; as EY said above, in a physical universe there are only quantum amplitudes, and if two agents have sufficiently accurate knowledge about the physical configuration of something, including their respective minds, they have to agree about that configuration, regardless of that they possibly have different values.

(2): I'm personally a bit confused about Eliezer's constant promotion of a language that de-subjectivizes morality. In most debates "objective" and "subjective" may entail a confusion when viewed in a naturalistic light; however, as I understand Eliezer's stance does boil down to a traditionally subjective viewpoint in the sense that it opposes the religious notion of morality as light shining down from the skies (and the notion of universally compelling arguments).

In regards to infallibility, an agent at most times has imperfect knowledge of right; I can't see how subjectivity entails infallibility. I don't even have perfect access to my current values, and there is also a huge set of moral arguments that would compel me to modify my current values if I heard them.

(3) The "why right means promoting X and Y" question is addressed by a recursive justification as discussed here and very specifically in the last paragraphs of Meaning of Right. If I ask "why should I do what is right?", that roughly means "why should I do what I should do?" or "why is right what is right?". I happen to be a mind that is compelled by a certain class of moral arguments, and I can reflect on this fact using my current mind, and, naturally, find that I'm compelled by a certain class of moral arguments.

EDIT: see also komponisto's comment.

Comment author: RichardChappell 01 February 2011 05:00:54PM 1 point [-]

re: infallibility -- right, the objection is not that you could infallibly know that XYZ is right. Rather, the problem is that you could infallibly know that your fundamental values are right (though you might not know what your fundamental values are).

Comment author: Kutta 02 February 2011 11:31:48AM *  3 points [-]

Rephrased, this knowledge is just the notion that you instantiate some computation instead of not doing (or being) anything. This way, my confidence in its truth is very high, although of course not 1.

Comment author: RichardChappell 02 February 2011 11:21:52PM 3 points [-]

We know we instantiate some computation. But it's a pre-theoretic datum that we don't know that our fundamental values are right. So EY's theory misdescribes the concept of rightness.

(This is basically a variation on Moore's Open Question Argument.)

Comment author: cousin_it 03 February 2011 12:16:36PM *  4 points [-]

are right

Huh?

I'd be okay with a strong AI that correctly followed my values, regardless of whether they're "right" by any other criterion.

If you think you wouldn't be okay with such an AI, I suspect the most likely explanation is that you're confused about the concept of "your values". Namely, if you yearn to discover some simple external formula like the categorical imperative and then enact the outcomes prescribed by that formula, then that's just another fact about your personal makeup that has to be taken into account by the AI.

And if you agree that you would be okay with such an AI, that means Eliezer's metaethics is adequate for its stated goal (creating friendly AI), whatever other theoretical drawbacks it might have.

Comment author: lukeprog 31 January 2011 01:15:55AM 2 points [-]

Thanks, Richard, for putting so much effort into your comment! When I find the time to parse this, I'll come back here to comment.

Comment author: lukeprog 09 March 2011 03:04:14AM 1 point [-]

Thinking more about this, it may have been better if Eliezer had not framed his meta-ethics sequence around "the meaning of right."

If we play rationalist's taboo with our moral terms and thus avoid moral terms altogether, what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.

His motivation for thinking this way is a thought experiment - which might become real in the relatively near future - about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn't object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar. But we would object to that scenario. Thus, what we care about seems not to be (a) but (b).

In a meta-ethicists terms, we could interpret Eliezer not as making an argument about the meaning of moral terms, but instead as making an argument that (b) is what gives us Reasons, not (a).

Now, all this meta-babble might not matter much. I'm pretty sure even if I was persuaded that the correct meta-ethical theory states that I should be okay with releasing a superintelligence that would rewire me to enjoy sitting in a jar, I would do whatever I could to prevent such a scenario and instead promote a superintelligence that would bring peace and joy and harmony and discovery and so on.

Comment author: Nisan 09 March 2011 03:25:19AM 1 point [-]

I thought being persuaded of a metaethical theory entails that whenever the theory tells you you should do X, you would feel compelled to do X.

Comment author: lukeprog 09 March 2011 03:56:42AM 2 points [-]

Only if motivational internalism is true. But motivational internalism is false.

Comment author: [deleted] 09 March 2011 04:00:07AM 0 points [-]

What's that?

Comment author: lukeprog 09 March 2011 04:41:07AM 0 points [-]
Comment author: [deleted] 09 March 2011 04:54:07AM 1 point [-]

I could get into how much I hate this kind of rejoinder if you bait me some more. I wasn't asking you for the number of acres in a square mile. Let me just rephrase:

I hadn't heard of motivational internalism before, could you expand your comment?

Comment author: [deleted] 09 March 2011 03:48:38AM 1 point [-]

This is a cool formulation. It's interesting that there are other things that can happen to you not similar to "being persuaded of a metaethical theory" that entail that whenever you are told to do X you're compelled to do X. (Voodoo or whatever.)

Comment author: Vladimir_Nesov 09 March 2011 11:53:08AM *  0 points [-]

what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.

His motivation for thinking this way is a thought experiment - which might become real in the relatively near future - about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn't object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar.

I don't see what plausible reasoning process could lead you to infer this unlikely statement (about motivation, given how many detail would need to be just right for the statement to happen to be true).

Also, even if you forbid modifying definition of human brain, things that initiate high-reward signals in our brains (or that we actually classify as "harmony" or "love") are very far from what we care about, just as whatever a calculator actually computes is not the same kind of consideration as the logically correct answer, even if you use a good calculator and aren't allowed sabotage. There are many reasons (and contexts) for reward in human brain to not be treated as indicative of goodness of a situation.

Comment author: lukeprog 09 March 2011 12:44:26PM *  0 points [-]

I don't understand your second paragraph. It sounds like you are agreeing to me, but your tone suggests you think you are disagreeing with me.

Comment author: Vladimir_Nesov 09 March 2011 04:02:49PM 1 point [-]

It was an explanation for why your thought experiment provides a bad motivation: we can just forbid modification of human brains to stop the thought experiment from getting through, but that would still leave a lot of problems, which shows that just this thought experiment is not sufficient motivation.

Comment author: lukeprog 09 March 2011 07:35:02PM 2 points [-]

Sure, the superintelligence thought experiment is not the fully story.

One problem with the suggestion of writing a rule to not alter human brains comes in specifying how the machine is not allowed to alter human brains. I'm skeptical about our ability to specify that rule in a way that does not lead to disastrous consequences. After all, our brains are being modified all the time by the environment, by causes that are on a wide spectrum of 'direct' and 'indirect.'

Other problems with adding such a rule are given here.

Comment author: Vladimir_Nesov 09 March 2011 08:22:03PM 2 points [-]

(I meant that subjective experience that evaluates situations should be specified using unaltered brains, not that brains shouldn't be altered.)

Comment author: lukeprog 09 March 2011 09:18:57PM 0 points [-]

You've got my curiosity. What does this mean? How would you realize that process in the real world?

Comment author: Vladimir_Nesov 10 March 2011 10:55:54AM 1 point [-]

Come on, this tiny detail isn't worth the discussion. Classical solution to wireheading, asking the original and not the one under the influence, referring to you-at-certain-time and not just you-concept that resolves to something unpredicted at any given future time in any given possible world, rigid-designator-in-time.

Comment author: lessdazed 02 July 2011 01:08:15PM 1 point [-]

What is objection (1) saying? That asserting there are moral facts is incompatible with the fact that people disagree about what they are? Specifically, when people agree that there is such a thing as a reason that applies to both of them, they disagree about how the reason is caused by reality?

Do we not then say they are both wrong about there being one "reason"?

I speak English(LD). You speak English(RC). The difference between our languages is of the same character as that between a speaker of Spanish and a speaker of French. I say "I" and you correctly read it as referring to lessdazed. You say "I" and I correctly read it as referring to RichardChapell. I have reasons(LD). You have reasons(RC). Do you think that were we perfect at monitoring what we each meant when we said anything and knew the relevant consequences of actions, the two of us would be capable of disagreeing when one of us asserted something in a sentence using the word "moral"? Why?

Or have I misread things?

Comment author: RichardChappell 02 July 2011 04:09:13PM *  3 points [-]

That asserting there are moral facts is incompatible with the fact that people disagree about what they are?

No, I think there are moral facts and that people disagree about what they are. But such substantive disagreement is incompatible with Eliezer's reductive view on which the very meaning of 'morality' differs from person to person. It treats 'morality' like an indexical (e.g. "I", "here", "now"), which obviously doesn't allow for real disagreement.

Compare: "I am tall." "No, I am not tall!" Such an exchange would be absurd -- the people are clearly just talking past each other, since there is no common referent for 'I'. But moral language doesn't plausibly function like this. It's perfectly sensible for one person to say, "I ought to have an abortion", and another to disagree: "No, you ought not to have an abortion". (Even if both are logically omniscient.) They aren't talking past each other. Rather, they're disagreeing about the morality of abortion.

Comment author: lessdazed 02 July 2011 04:59:41PM 2 points [-]

But moral language doesn't plausibly function like this.

It's not plausible(RC, 7/1/2011 4:25 GMT), but it is plausible(LD, 7/1/2011 4:25 GMT).

Compare: "I am tall." "No, I am not tall!" Such an exchange would be absurd -- the people are clearly just talking past each other, since there is no common referent for 'I'.

It's not impossible for people to be confused in exactly such a way.

It's perfectly sensible for one person to say, "I ought to have an abortion", and another to disagree: "No, you ought not to have an abortion". (Even if both are logically omniscient.)

That's begging the question.

That intuition pump imagines intelligent people disagreeing, finds it plausible, notices that intelligent people disagreeing proves nothing, then replaces the label "intelligent" with "omniscient" (since that, if proven, would prove something) without showing the work that would make the replacement valid. If the work could be shown, the intuition pump wouldn't be very valuable, as one could just use the shown work for persuasion rather than the thought experiment with the disagreeing people. I strongly suspect that the reason the shown work is unavailable is because it does not exist.

Eliezer's reductive view on which the very meaning of 'morality' differs from person to person.

Forget morality for one second. Doesn't the meaning of the word "hat" differ from person to person?

It's perfectly sensible for one person to say, "I ought to have an abortion"

It's only sensible to say if/because context forestalls equivocation (or tries to, anyway). Retroactively removing the context by coming in the conversation with a different meaning of ought (even if the first meaning of "ought" was "objective values, as I think they are, as I think I want them to be, that are universally binding on all possible minds, and I would maintain under any coherent extrapolation of my values" where the first person is wrong about those facts and the second meaning of "ought" is the first person's extrapolated volition) introduces equivocation. It's really analogous to saying "No, I am not tall".

Where the first person says "X would make me happy, I want to feel like doing X, and others will be better off according to balancing equation Y if I do X, and the word "ought" encompasses when those things coincide according to objective English, so I ought to do X", and the second person says "X would make you happy, you want to feel like doing X, and others will not be better off according to balancing equation Z if you do X, and the word "ought" encompasses when those things coincide according to objective English, so you ought not do X", they are talking past each other. Purported debates about the true meaning of "ought" reveal that everyone has their own balancing equation, and the average person thinks all others are morally obliged by objective morality to follow his or her equation. In truth, the terms "make happy" and want to feel like doing" are rolled up into the balancing equation, but in it (for Westerners) terms for self and others seem as if they are of different kind.

Comment author: RichardChappell 03 July 2011 04:05:03PM 2 points [-]

Purported debates about the true meaning of "ought" reveal that everyone has their own balancing equation, and the average person thinks all others are morally obliged by objective morality to follow his or her equation.

You're confusing metaethics and first-order ethics. Ordinary moral debates aren't about the meaning of "ought". They're about the first-order question of which actions have the property of being what we ought to do. People disagree about which actions have this property. They posit different systematic theories (or 'balancing equations' as you put it) as a hypothesis about which actions have the property. They aren't stipulatively defining the meaning of 'ought', or else their claim that "You ought to follow the prescriptions of balancing equation Y" would be tautological, rather than a substantive claim as it is obviously meant to be.

Comment author: lessdazed 03 July 2011 05:37:41PM 0 points [-]

Ordinary moral debates aren't about the meaning of "ought".

I know that, which is why I said "Purported debates about the true meaning of 'ought'" rather than "ordinary debates, which are about the true meaning of 'ought'".

They're about the first-order question of which actions have the property of being what we ought to do. People disagree about which actions have this property.

Please be careful not to beg the question. People agree that there is such a property, but that is something about which they can be wrong.

They aren't stipulatively defining the meaning of 'ought', or else their claim that "You ought to follow the prescriptions of balancing equation Y" would be tautological...

Rather, they aren't trying to stipulatively define the meaning of 'ought', or else their claim that "You ought to follow the prescriptions of balancing equation Y" would be tautological.

In fact, due to people's poor self-insight, time limits, and the sometimes over-coarse granularity of language, they do not stipulate their actual balancing equation. Had they perfect insight and ability to represent their insights, it would be such a tautology. They would cease to speak like that had they the additional insight that for it to do the work it is called upon to do,"ought" is a word that needs grounding in the context of the real reasons for action of beings More generally, they are speaking an idiolect even regarding other definitions.

...rather than a substantive claim as it is obviously meant to be.

It's meant to be such a claim, but it is in error because the speaker is confused about morality, and in a sense is not even wrong. They are claiming some actions have an objective moral valuation binding upon all intelligent beings, but they may as well claim the action has the property of being a square circle - or better yet, a perfect circle for pi is exactly 3, which is something I have witnessed a religious person claim is true.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I don't understand either why you believe as you do or what good justification you might have for it.

I can see why one might want to make truth claims in which it falls out the folk have the least amount of confusion to be embarrassed about and are least wrong, and if one begins with the assumption that there are "moral facts" in the strongest sense, that's a good start. However, that neither prevents one from having to say they are wrong about an enormous amount nor does it prevent one from having to claim that others, such as me, are wrong about moral facts. I can also see how it might be comforting were moral facts real in that sense and how the world might be better off if people believe in them, but obviously those things would not make them true.

At least you seem to understand the position you are arguing against and I am advocating. You repeatedly have tried to dissuade me from it by pointing out extensions of my position and counting on me to disclaim it from reductio ad absurdum. But it always seems to be by appealing to the same notion, that people simply cannot be wrong about some sorts of things they believe, which is a position I reject.

So, assuming I bite the bullet and don't find it implausible that some people are fundamentally confused when they make moral claims, what other reasons might there be to agree with them? Why should I believe that intensionally defined "objective morality" has any extension at all?

Comment author: RichardChappell 03 July 2011 05:57:30PM *  1 point [-]

I'm not arguing for moral realism here. I'm arguing against metaethical reductionism, which leaves open either realism OR error theory.

For all I've said, people may well be mistaken when they attribute normative properties to things. That's fine. I'm just trying to clarify what it is that people are claiming when they make moral claims. This is conceptual analysis, not metaphysics. I'm pointing out that what you claim to be the meaning of 'morality' isn't what people mean to be talking about when they engage in moral discourse. I'm not presupposing that ordinary people have any great insight into the nature of reality, but they surely do have some idea of what their own words mean. Your contrary linguistic hypothesis seems completely unfounded.

Comment author: lessdazed 03 July 2011 06:05:51PM 1 point [-]

I'm pointing out that what you claim to be the meaning of 'morality' isn't what people mean to be talking about when they engage in moral discourse.

When I was young, I learned that the tooth fairy was really my mother all along.

What do you think of that?

(This isn't meant to be insulting or anything similar.)

Comment author: RichardChappell 03 July 2011 09:21:39PM *  1 point [-]

No, you learned that the tooth fairy doesn't exist, and that your mother was instead responsible for the observable phenomena that you had previously attributed to the tooth fairy.

(It's a good analogy though. I do think that claiming that morality exists "as a computation" is a lot like claiming that the tooth fairy really exists "as one's mother".)

Comment author: lessdazed 04 July 2011 10:13:07AM 6 points [-]

you learned that the tooth fairy doesn't exist

Yes.

No, you learned that the tooth fairy doesn't exist

No.

Usually, the first thing to do when guessing about a random number from 1-100 is to split the possibilities in half by asking if it is more than 50 (or odd. Etc.)

The tooth fairy example gets a variety of responses, from people insisting it is just objectively wrong to say "the tooth fairy doesn't exist" to those saying it is just objectively wrong to say the tooth fairy was really my mother. I happen to agree with you about what the best way is to describe what went on in this specific case. However, this is a standard blegg-rube situation that is unusual only in that it is not clear which way is best to describe the phenomenon to others.

There is a constellation of phenomena that correlate to each other - the fairy being female, being magic, having diaphanous wings, collecting things for money, those things being stored under pillows, those things being teeth. None of these is qualitatively essential to be a tooth fairy to most people than "having ten fingers" is essential to being human. If tomorrow we learn that magic is real, a female sprite collects teeth from under pillows, and does so on the back of a termite (and has size-changing technology/magic, why not?), most people would naively say "the tooth fairy does not fly, but burrows on the back of a termite". That's OK, but not great if the true nature of the situation is not recognized, and they fall into error if they think "tooth fairy" has a meaning divorced from flight. Likewise, those who say "there was never a 'tooth fairy', there is rather the 'burrowing tooth fairy'" are right that there was never a thing exactly like the classic description, but this group makes an error if they demand the first stop calling the "burrowing tooth fairy" the "tooth fairy".

There is more to say, an individual who makes up explanations ad hoc is not communicating, and the relative confluence of idiolects is valid because of the tinkerbell effect. that makes saying "No, you learned that the tooth fairy doesn't exist" really peculiar, in the face of the many who hold the opposite position, likewise endorsed as language by common usage!

When it comes to "morality", the constellation of things represented with a token labeled "second order morality" had several stars, significantly organized around a few (mutually contradictory, but all validly denoted by the word "morality") sets of answers to these eight questions in the SEP on Moral Reasoning:

How do relevant considerations get taken up in moral reasoning? Is it essential to moral reasoning for the considerations it takes up to be crystallized into, or ranged under, principles? How do we sort out which moral considerations are most relevant? In what ways do motivational elements shape moral reasoning? What is the best way to model the kinds of conflicts among considerations that arise in moral reasoning? Does moral reasoning include learning from experience and changing one's mind? How can we reason, morally, with one another? What is the importance of institutional authority and social roles to the structure of moral reasoning?

One characteristic of things often (and validly) called "morality" is that they include the following among other pieces to answer the above questions: "'That which is good' is an ephemeral property that is infused in things in a dualistic way, it can be separate from all concerns beings may have, "good" can be binding in a way that is tautological and pointless...etc.

Reductionism continues the tradition of "morality", like other words, not meaning exactly what it did in the past, and it may be better to think of it as error theory in some cases. But all new theories claim the ones they supersede are in error, and there is nothing special about "morality minus magical deontology" or "moral reductionism" that one should consider them bad English by the usual standard.

After typing all of this, it strikes me that cars are a better example of no one component being essential. Cars without wheels will be "flying cars", unless they get sufficient lift and air etc. such that as one goes down the line of models from flying jalopy to sports flying car, each successive flying car will be called "car" by fewer and fewer people as it more resembles a fighter jet or so...as perhaps our descendants will call nuclear-powered flying vehicles.

The important thing is that reductionism gets it right when describing reality, the problems with deontology etc., and explains why people think erroneously, and it then proceeds to tell one how to act such that first order moral claims have meaning. If despite all that, you want to focus on its erosion of folk theories to call it an error theory, that is understandable, but it will be uninteresting if you say to the multitude "It is wrong(RC) to call reductionism 'moral'(RC)," since it is not wrong(everybody who recognizes not to misuse language by fixating on a connotation that often went along with a word when the contents described by that word are otherwise vindicated and under discussion, and we're discussing language which means 1) constelations and 2) tinkerbell).

Comment author: Peterdjones 03 July 2011 09:40:47PM *  0 points [-]

That depends whether you are going to replace objective morality with subective morality or with error theory. It can be argued that subjectivism is an unstable position that amount to error theory.

Comment author: orthonormal 01 February 2011 05:40:24PM 1 point [-]

(3) Just as simple subjectivism is an implausible theory of what 'right' means, so Eliezer's meta-semantic subjectivism is an implausible theory of why 'right' means promoting external goods X, Y, Z. An adequately objective metaethics shouldn't even give preferences a reference-fixing role.

This seems to me like begging the question. Can you expand on this?