You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

[Link] An attempt in layman's language to explain the metaethics sequence in a single post.

0 Post author: Bound_up 12 October 2016 01:57PM

Comments (17)

Comment author: DanArmak 12 October 2016 02:55:20PM 2 points [-]

Without commenting on whether this presentation matches the original metaethics sequence (with which I disagree), this summary argument seems both unsupported and unfalsifiable.

  1. No evidence is given for the central claim, that humans can and are converging towards a true morality we would all agree about if only we understood more true facts.
  2. We're told that people in the past disagreed with us about some moral questions, but we know more and so we changed our minds and we are right while they were wrong. But no direct evidence is given for us being more right. The only way to judge who's right in a disagreement seems to be "the one who knows more relevant facts is more right" or "the one who more honestly and deeply considered the question". This does not appear to be an objectively measurable criterion (to say the least).
  3. The claim that ancients, like Roman soldiers, thought slavery was morally fine because they didn't understand how much slaves suffer is frankly preposterous. Roman soldiers (and poor Roman citizens in general) were often enslaved, and some of them were later freed (or escaped from foreign captivity). Many Romans were freedmen or their descendants - some estimate that by the late Empire, almost all Roman citizens had at least some slave ancestors. And yet somehow these people, who both knew what slavery was like and were often in personal danger of it, did not think it immoral, while white Americans in no danger of enslavement campaigned for abolition.
Comment author: hairyfigment 14 October 2016 11:09:12AM 0 points [-]

I'm getting really sick of this claim that Eliezer says all humans would agree on some morality under extrapolation. That claim is how we get garbage like this. At no point do I recall Eliezer saying psychopaths would definitely become moral under extrapolation. He did speculate about them possibly accepting modification. But the paper linked here repeatedly talks about ways to deal with disagreements which persist under extrapolation:

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted. (emphasis added)

Coherence is not a simple question of a majority vote. Coherence will reflect the balance, concentration, and strength of individual volitions. A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity. The variables are quantitative, not qualitative.

(Naturally, Eugine Nier as "seer" downvoted all of my comments.)

The metaethics sequence does say IMNSHO that most humans' extrapolated volitions (maybe 95%) would converge on a cluster of goals which include moral ones. It furthermore suggests that this would apply to the Romans if we chose the 'right' method of extrapolation, though here my understanding gets hazier. In any case, the preferences that we would loosely call 'moral' today, and that also survive some workable extrapolation, are what I seem to mean by "morality".

One point about the ancient world: the Bhagavad Gita, produced by a warrior culture though seemingly not by the warrior caste, tells a story of the hero Arjuna refusing to fight until his friend Krishna convinces him. Arjuna doesn't change his mind simply because of arguments about duty. In the climax, Krishna assumes his true form as a god of death with infinitely many heads and jaws, saying, 'I will eat all of these people regardless of what you do. The only deed you can truly accomplish is to follow your warrior duty or dharma.' This view seems plainly environment-dependent.

Comment author: Bound_up 12 October 2016 05:54:11PM 0 points [-]

You're right; I've provided no evidence.

Do you think the idea is sufficiently coherent and non-self-contradictory that the way to find out if it's right or wrong is to look for evidence?

If it was incoherent or contradicted itself, it wouldn't even need evidence to be disproven; we would already know it's wrong. Have I avoided being wrong in that way?

(by the way, understanding slavery might be necessary, but not sufficient to get someone to be against it. They might also need to figure out that people are equal, too. Good point, I might need to add that note into the post).

Comment author: Lumifer 12 October 2016 06:05:49PM *  1 point [-]

Do you think the idea is sufficiently coherent and non-self-contradictory that the way to find out if it's right or wrong is to look for evidence?

You do understand that debates about objective vs relative morality has been going on for millenia?

They might also need to figure out that people are equal, too

No, they don't if they themselves are in danger of becoming slaves. Notably, a major source of slaves in the Ancient world was defeated armies. Slaves weren't clearly different people (like the blacks were in America), anyone could become a slave if his luck turned out to be really bad.

Comment author: Bound_up 12 October 2016 06:32:26PM 0 points [-]

Right. Someone could be against slavery for THEM personally without being against slavery in general if they didn't realize that what was wrong for them was also wrong for others. That's all I'm getting at, there.

Or do you mean that they should have opposed slavery for everybody as a sort of game theory move to reduce their chance of ever becoming a slave?

"You do understand that debates about objective vs relative morality has been going on for millenia?"

What I'm getting at here is that most moral theories are so bad you don't even need to talk about evidence. You can show them to be wrong just because they're incoherent or self-contradictory.

It's a pretty low standard, but I'm asking if this theory is at least coherent and consistent enough that you have to look at evidence to know if it's wrong, instead of just pointing at its self-defeating nature to show it's wrong. If so, yay, it might be the best I've ever seen. :)

Comment author: Lumifer 12 October 2016 07:07:50PM 1 point [-]

Someone could be against slavery for THEM personally without being against slavery in general if they didn't realize that what was wrong for them was also wrong for others.

Huh? I'm against going to jail personally without being against the idea of jail in general. In any case, wasn't your original argument that ancient Greeks and Romans just didn't understand what does it mean to be a slave? That clearly does not hold.

most moral theories are so bad you don't even need to talk about evidence. You can show them to be wrong just because they're incoherent or self-contradictory.

Do you mean descriptive or prescriptive moral theories? If descriptive, humans are incoherent and self-contradictory.

Which moral theories do you have in mind? A few examples will help.

Comment author: Bound_up 12 October 2016 10:18:19PM 0 points [-]

Mmm, that's not quite the right abstraction. You're probably against innocents going to jail in general, no?

Whereas some Roman might not care, as long as it's no one they care about.

All I'm getting at is that the Romans didn't think certain things were wrong, but if they were shown in a sufficiently deep way everything we know, they would be moved by it, whereas if we were shown everything they know, we would not find it persuasive of their position. Neither would they, after they had seen what we've seen.

I'm talking metaethics, what makes something moral, what it means for something to be moral. Failed ones include divine command theory, the "whatever contributes to human flourishing" idea, whatever makes people happy, whatever matches some platonic ideals out there somehow, whatever leads to selfish interest, etc.

Comment author: Lumifer 13 October 2016 02:24:26PM *  0 points [-]

if they were shown in a sufficiently deep way everything we know, they would be moved by it

That doesn't seem obvious to me at all.

Let's try it on gay marriage. Romans certainly knew and practiced homosexuality, same for marriage. What knowledge exactly do you want to convey to them to persuade them that gay marriage is a good thing?

I'm talking metaethics, what makes something moral

So, prescriptive. I am not sure in which way do you consider the theories "failed" -- in the sense that they have not risen to the status of physics meaning being able to empirically prove all their claims? That doesn't look to be a viable criterion. In the sense of not having taken over the world? I don't know, the divine command theory is (or, at least, has been) pretty good at that. You probably wouldn't want a single theory to take over the world, anyway.

Comment author: DanArmak 12 October 2016 10:13:13PM 0 points [-]

Do you think the idea is sufficiently coherent and non-self-contradictory that the way to find out if it's right or wrong is to look for evidence?

Yes, I think it is coherent.

Ideological Turing test: I think your theory is this: there is some set of values, which we shall call Morals. All humans have somewhat different sets of lower-case morals. When people make moral mistakes, they can be corrected by learning or internalizing some relevant truths (which may of course be different in each case). These truths can convince even actual humans to change their moral values for the better (as opposed to values changing only over generations), as long as these humans honestly and thoroughly consider and internalize the truths. Over historical time, humans have approached closer to true Morals, and we can hope to come yet closer, because we generally collect more and more truths over time.

the way to find out if it's right or wrong is to look for evidence?

If you mean you don't have any evidence for your theory yet, then how or why did you come by this theory? What facts are you trying to explain or predict with it?

Remember that by default, theories with no evidence for them (and no unexplained facts we're looking for a theory about) shouldn't even rise to the level of conscious consideration. It's far, far more likely that if a theory like that comes to mind, it's for due to motivated reasoning. For example, wanting to claim your morality is better by some objective measure than that of other people, like slavers.

by the way, understanding slavery might be necessary, but not sufficient to get someone to be against it. They might also need to figure out that people are equal, too.

That's begging the question. Believing that "people are equal" is precisely the moral belief that you hold and ancient Romans didn't. Not holding slaves is merely one of many results of having that belief; it's not a separate moral belief.

But why should Romans come to believe that people are equal? What sort of factual knowledge could lead someone to such a belief, despite the usually accepted idea that should cannot be derived from is?

Comment author: Bound_up 12 October 2016 11:16:29PM 0 points [-]

This is an explanation of Yudkowsky's idea from the metaethics sequence. I'm just trying to make it accessible in language and length with lots of concept handles and examples.

Technically, you could believe that people are equally allowed to be enslaved. All people equal + it's wrong to make me a slave = it's wrong to make anyone a slave.

"All men are created equal" emerges from two or more basic principles people are born with. You might say: "Look, you have value, yah? And your loved ones? Would they stop having value if you forgot about them? No? They have value whether or not you know them? How did you conclude they have value? Could that have happened with other people, too? Would you then think they had value? Would they stop having value if you didn't know them? No? Well, you don't know them; do they have value?

You take "people I care about have value" (born with it) and combine it with "be consistent" (also born with), and you get "everyone has value."

That's the idea in principle, anyway. You take some things people are all born with, and they combine to make the moral insights people can figure out and teach each other, just like we do with math.

Comment author: DanArmak 13 October 2016 09:47:14AM 1 point [-]

Technically, you could believe that people are equally allowed to be enslaved.

In a sense, the ancient Romans did believe this. Anyone who ended up in the same situation - either taken as a war captive or unable to pay their debts - was liable to be sold as a slave. So what makes you think your position is objectively better than theirs?

"All men are created equal" emerges from two or more basic principles people are born with. You might say: "Look, you have value, yah? And your loved ones? Would they stop having value if you forgot about them? No? They have value whether or not you know them? How did you conclude they have value? Could that have happened with other people, too? Would you then think they had value? Would they stop having value if you didn't know them? No? Well, you don't know them; do they have value?

This assumes without argument that "value" is something people intrinsically have or can have. If instead you view value as value-to-someone, i.e. I value my loved ones, but someone else might not value them, then there is no problem.

And it turns out that yes, most people did not have an intuition that anyone has intrinsic value just by virtue of being human. Most people throughout history assigned value only to ingroup members, to the rich and powerful, and to personally valued individuals. The idea that people are intrinsically valuable is historically very new, still in the minority today globally, and for both these reasons doesn't seem like an idea everyone should naturally arrive at if they only try to universalize their intuitions a bit.

Comment author: TheAncientGeek 13 October 2016 05:06:17PM 0 points [-]

Technically, you could believe that people are equally allowed to be enslaved. All people equal + it's wrong to make me a slave = it's wrong to make anyone a slave.

You realise that's a reinvention of Kant?

Comment author: TheAncientGeek 12 October 2016 04:05:54PM *  1 point [-]

Unpacking "should" as " morally obligated to" is potentially helpful, so inasmuch as you can give separate accounts of "moral" and "obligatory".

The elves are not moral. Not just because I, and humans like me happen to disagree with them, no, certainly not. The elves aren’t even trying to be moral. They don’t even claim to be moral. They don’t care about morality. They care about “The Christmas Spirit,” which is about eggnog and stuff

That doesn't generalise to the point that non humans have no morality. You have made things too easy on yourself by having the elves concede that the Christmas spirit isn't morality. You need to to put forward some criteria for morality and show that the Christmas Spirit doesn't fulfil them. (One of the odd things about the Yudkowskian theory is that he doesnt feel the need to show that human values are the best match to some pretheoretic botion of morality, he instead jumps straight to the conclusion).

The hard case would be some dwarves, say, who have a behavioural code different from our own, and who haven't conceded that they are amoral. Maybe they have a custom whereby any dwarf who hits a rich seam of ore has to raise a cry to let other dwarves have a share, and any dwarf who doesn't do this is criticised and shunned. If their code of conduct passed the duck test .. is regarded as obligatory, involves praise and blame, and so on ... why isn't that a moral system?

This is so weird to them that they’d probably just think of it as…ehh, what? Just weird. They couldn’t care less. Why on earth would they give food to millions of starving children? What possible reason…who even cares?

If they have failed to grasp that morality is obligatory, have they understood it at all? They might continue caring more about eggnog, of course. That is beside the point... morality means what you should care about, not what you happen to do.

Morality needs to be motivating, and rubber stamping your existing values as moral achieves that, but being motivating is not sufficient. A theory of morality also needs to be able to answer the Open Question objection, meaning in this case, the objection that it is not obvious that you should value something just because you do.

So, to say the elves have their own “morality,” is not quite right. The elves have their own set of things that they care about instead of morality

That is arguing from the point that morality is a label for whatever humans care about, not toward it.

This helps us see the other problem, when people say that “different people at different times in history have been okay with different things, who can This is so weird to them that they’d probably just think of it as…ehh, what? Just weird. They couldn’t care less. Why on earth would they give food to millions of starving children? What possible reason…who even cares? who’s really right?”

There are many ways of refuting relativism, and most don't involve the claim that humans are uniquely moral.

Morality is a fixed thing. Frozen, if you will. It doesn’t change.

It is human value, or it is fixed.. choose one. Humans have valued many different things. One of the problems with the rubber stamping approach is that things the audience will see as immoral such as slavery and the subjugation of women have been part of human value.

Rather, humans change. Humans either do or don’t do the moral thing. If they do something else, that doesn’t change morality, but rather, it just means that that human is doing an immoral

If that is true, then you need to stop saying that morality is human values. and start saying morality is human values at time T. And justify the selection of time, etc. And even at that, you won't support your other claims. because what you need to prove is that morality is unique, that only one thing can fulfil the role.

Rather, humans happen to care about moral things. If they start to care about different things, like slavery, that doesn’t make slavery moral, it just means that humans have stopped caring about moral things.

If it is possible for human values to diverge from morality. then something else must define morality, because human values can't diverge from human values. So you are not using a stipulative definition... here....although you are when you argue that elves can't be moral. Here, you and Yudkowsky have noticed that your theory entails the same problem as relativism: if morality is whatever people value, and if what people happen to value is intuitively immoral , slavery, torture,whatever, then there's no fixed standard of morality. The label "moral" has been placed on a moving target. (Standard relativism usually has this problem synchronously , ie different communities are said to have different but equally valid moralities at the same time, but it makes little difference if you are asserting that the global community has different but equally valid moralities at different times)

So, when humans disagree about what’s moral, there’s a definite answer.

There is from many perspectives , but given that human values can differ, you get no definite answer by defining morality as human value. You can avoid the problems of relativism by setting up an external standard, and there are many theories of that type, but they tend to have the problem that the external standard is not naturalistic....God's commands, the Form of the good, and so on. I think Yudkowsky wants a theory that is non arbitrary and also naturalistic. I don't think he arrives a single theory that does both. If the Moral Equation is just a label for human intuition, then it ssuffers from all the vagaries of labeling values as moral, the original theory.

How do we find that moral answer, then? Unfortunately, there is no simple answer

Why doesn't that constitute an admission that you don't actually have a theory of morality?

You see, we don’t know all the pieces of morality, not so we can write them down on paper. And even if we knew all the pieces, we’d still have to weigh which ones are worth how much compared to each other.

On the assumption that all human value gets thrown into the equation, it certainly would be complex. But not everyone has that problem. since people have criteria for somethings being moral , and others but being. which simplify the equation. and allow you to answer the questions you were struggling with above. You know, you don't have to pursue assumptions to their illogical conclusions.

Humans all care about the same set of things (in the sense I’ve been talking about). Does this seem contradictory? After all, we all know humans do not agree about what’s right and wrong; they clearly do not all care about the same things.

On the face of it , it's contradictory. There maybe something else that is smooths out the contradictions, such as the Moral Equation, but that needs justification of its own.

Well, they do. Humans are born with the same Morality Equation in their brains, with them since birth.

Is that a fact? It's eminently naturalistic, but the flip side to that is that it is, therefore, empirically refutable. If an individual's Morality Equation is just how their moral intuition works, then the evidence indicates that intuitions can vary enough to start a war or two. So the Morality Equation appears not to be conveniently the same in everybody.

How then all their disagreements? There are three ways for humans to disagree about morals, even though they’re all born with the same morality equation in their heads (1 Don't do it, 2 don't do it right, 3 don't want to do it)

What does it mean to do it wrong, if the moral equation is just a label for black box intuitive reasoning? If you had an external standard, as utilitarians and others do, then you could determine whose use of intuition is right use according to it. But in the absence of an external standard, you could have a situation where both parties intuit differently, and both swear they are taking all factors into account. Given such a stalemate, how do you tell who is right? It would be convenient if the only variations to the output of the Morality Equation were caused by variations in the input, but you cannot assume something is true just because it would be convenient.

If the Moral Equation is something ideal and abstract, why can't aliens partake? That model of ethics is just what s needed to explain how you can have multiple varieties of object level morality that actually all are morality: different values fed into the same equation produce different results, so object level morality varies although the underlying principle us the same..

Comment author: Bound_up 12 October 2016 06:02:16PM 0 points [-]

Okay. By saying "If they have failed to grasp that morality is obligatory, have they understood it at all? They might continue caring more about eggnog, of course. That is beside the point... morality means what you should care about, not what you happen to do."

it seems you have not understood the idea. Were there any parts of the the post that seemed unclear that you think I might make clearer?

Because the whole point is that to say something is moral = you should do it = it is valued according to the morality equation.

For an Elf to agree something is moral is also to agree that they should do it. When I say they agree it's moral and don't care, that also means they agree they should do it and don't care.

Something being Christmas Spiritey = you Spiritould do it. Humans might agree that something is Christmas Spirit-ey, and agree that they spiritould do it, they just don't care about what they spiritould do, they only care about what they should do.

moral is to Christmas spiritey what "should" is to (make up a word like) "spiritould"

Obligatory is just a kind of "should." Elves agree that some things are obligatory, and don't care, they care about what's ochristmastory.

.

Likewise, to say that today's morality equation is the "best" is to say that today's morality equation is the equation which is most like today's morality equation. Tautology.

Best = most good, and good = valued by the morality equation.

Comment author: TheAncientGeek 13 October 2016 06:30:32PM *  0 points [-]

it seems you have not understood the idea. Were there any parts of the the post that seemed unclear that you think I might make clearer?

Almost everything. You explain morality by putting forward one theory. Under those circumstances, most people would expect to see some critique of other theories, and explanation of why your theory is the One True Theory. You don't do the first, and it is not clear that you are even trying to do the second.

Because the whole point is that to say something is moral = you should do it = it is valued according to the morality equation.

And to say that only humans have morality. But if there is something the Elves should do, then morality applies to them., contradicting that claim.

For an Elf to agree something is moral is also to agree that they should do it. When I say they agree it's moral and don't care, that also means they agree they should do it and don't care.

That doesn't help. For one thing, humans don't exactly want to be moral...their moral fibre has to be buttressed bty various punishments and rewards. For another "should" and "want to" are not synonyms..but "moral" and "what you should do" are. So if there is something the Elves should do, at that point you have established that morality applies to the Elves, and the fact that they don't want to do it is a side-issue. (And of course they could tweak their own motivations by constructing punishments and rewards).

Something being Christmas Spiritey = you Spiritould do it. Humans might agree that something is Christmas Spirit-ey, and agree that they spiritould do it, they just don't care about what they spiritould do, they only care about what they should do.

OK. Now you seem to be saying..without quite making it quite explicit of course, ..that morality is by definition unique to humans, because the word "moral" just labels what motivates humans, in the way that "Earth" or "Terra" labels the planet where humans live. That claim isn't completely incomprehensible, it's just strange and arbitrary, and what is considerably strange is the way you feel no need to defend it against alternative theories -- the main alternative being that morality is multiply instantiable, that other civilisations could have their own versions. like they have their own versions , in the way they could have their own versions of houses or money.

You state it as though it is obvious, yet it has gone unnoticed for thousands of years.

Suppose I were to announce that dark matter is angels' tears. Doesn't it need some expansion? That's how your claim reads, that' the outside view.

Obligatory is just a kind of "should." Elves agree that some things are obligatory, and don't care, they care about what's ochristmastory.

Obligatory is a kind of "should" *that shouldn't be overridden by other considerations. (A failure to do what is obligatory is possible, of course, but it is important to remember that it is seen as a lapse, as something wrong, not a valid choice). Yet the Elves are overriding it, casting doubt on whether they have actually understood the concept of "obligatory"

Likewise, to say that today's morality equation is the "best" is to say that today's morality equation is the equation which is most like today's morality equation. Tautology.

Since anyone can say that at any time, that breaks the meaning of "best", which is supposed to pick out something unique. That would be a reductio ad absurdum of your own theory.

Comment author: entirelyuseless 13 October 2016 02:15:44AM 0 points [-]

I think it is perfectly obvious that this usage of "should" and so on is wrong. A paperclipper believes that it should make paperclips, and it means exactly the same thing by "should" that I do when I say I should not murder.

And when I say it is obvious, I mean it is obvious in the same way that it is obvious that you are using the word "hat" wrong if you use it for a coat.

Comment author: BiasedBayes 13 October 2016 11:45:12AM 0 points [-]

Morality binds and blinds. People derive moral claims from emotional and intuitive notions. It can feel good and moral to do amoral things. Objective morality has to be tied to evidence what really is human wellbeing; not to moral intuitions that are adaptions to the benefit of ones ingroup; or post hoc thought experiments about knowledge.