LESSWRONG
LW

by Eliezer Yudkowsky
7 min read
119

25

Followup to:  Is Fairness Arbitrary?, Joy in the Merely Good,  Sorting Pebbles Into Correct Heaps

Yesterday, I presented the idea that when only five people are present, having just stumbled across a pie in the woods (a naturally growing pie, that just popped out of the ground) then it is fair to give Dennis only 1/5th of this pie, even if Dennis persistently claims that it is fair for him to get the whole thing.  Furthermore, it is meta-fair to follow such a symmetrical division procedure, even if Dennis insists that he ought to dictate the division procedure.

Fair, meta-fair, or meta-meta-fair, there is no level of fairness where you're obliged to concede everything to Dennis, without reciprocation or compensation, just because he demands it.

Which goes to say that fairness has a meaning beyond which "that which everyone can be convinced is 'fair'".  This is an empty proposition, isomorphic to "Xyblz is that which everyone can be convinced is 'xyblz'".  There must be some specific thing of which people are being convinced; and once you identify that thing, it has a meaning beyond agreements and convincing.

You're not introducing something arbitrary, something un-fair, in refusing to concede everything to Dennis.  You are being fair, and meta-fair and meta-meta-fair.  As far up as you go, there's no level that calls for unconditional surrender.  The stars do not judge between you and Dennis—but it is baked into the very question that is asked, when you ask, "What is fair?" as opposed to "What is xyblz?"

Ah, but why should you be fair, rather than xyblz?  Let us concede that Dennis cannot validly persuade us, on any level, that it is fair for him to dictate terms and give himself the whole pie; but perhaps he could argue whether we should be fair?

The hidden agenda of the whole discussion of fairness, of course, is that good-ness and right-ness and should-ness, ground out similarly to fairness.

Natural selection optimizes for inclusive genetic fitness.  This is not a disagreement with humans about what is good.  It is simply that natural selection does not do what is good: it optimizes for inclusive genetic fitness.

Well, since some optimization processes optimize for inclusive genetic fitness, instead of what is good, which should we do, ourselves?

I know my answer to this question.  It has something to do with natural selection being a terribly wasteful and stupid and inefficient process.  It has something to do with elephants starving to death in their old age when they wear out their last set of teeth.  It has something to do with natural selection never choosing a single act of mercy, of grace, even when it would cost its purpose nothing: not auto-anesthetizing a wounded and dying gazelle, when its pain no longer serves even the adaptive purpose that first created pain.  Evolution had to happen sometime in the history of the universe, because that's the only way that intelligence could first come into being, without brains to make brains; but now that era is over, and good riddance.

But most of all—why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good?  What is even the appeal of this, morally or otherwise?  At all?  I know people who claim to think like this, and I wonder what wrong turn they made in their cognitive history, and I wonder how to get them to snap out of it.

When we take a step back from fairness, and ask if we should be fair, the answer may not always be yes.  Maybe sometimes we should be merciful.  But if you ask if it is meta-fair to be fair, the answer will generally be yes.  Even if someone else wants you to be unfair in their favor, or claims to disagree about what is "fair", it will still generally be meta-fair to be fair, even if you can't make the Other agree.  By the same token, if you ask if we meta-should do what we should, rather than something else, the answer is yes.  Even if some other agent or optimization process does not do what is right, that doesn't change what is meta-right.

And this is not "arbitrary" in the sense of rolling dice, not "arbitrary" in the sense that justification is expected and then not found.  The accusations that I level against evolution are not merely pulled from a hat; they are expressions of morality as I understand it.  They are merely moral, and there is nothing mere about that.

In "Arbitrary" I finished by saying:

The upshot is that differently structured minds may well label different propositions with their analogues of the internal label "arbitrary"—though only one of these labels is what you mean when you say "arbitrary", so you and these other agents do not really have a disagreement.

This was to help shake people loose of the idea that if any two possible minds can say or do different things, then it must all be arbitrary.  Different minds may have different ideas of what's "arbitrary", so clearly this whole business of "arbitrariness" is arbitrary, and we should ignore it.  After all, Sinned (the anti-Dennis) just always says "Morality isn't arbitrary!" no matter how you try to persuade her otherwise, so clearly you're just being arbitrary in saying that morality is arbitrary.

From the perspective of a human, saying that one should sort pebbles into prime-numbered heaps is arbitrary—it's the sort of act you'd expect to come with a justification attached, but there isn't any justification.

From the perspective of a Pebblesorter, saying that one p-should scatter a heap of 38 pebbles into two heaps of 19 pebbles is not p-arbitrary at all—it's the most p-important thing in the world, and fully p-justified by the intuitively obvious fact that a heap of 19 pebbles is p-correct and a heap of 38 pebbles is not.

So which perspective should we adopt?  I answer that I see no reason at all why I should start sorting pebble-heaps.  It strikes me as a completely pointless activity.  Better to engage in art, or music, or science, or heck, better to connive political plots of terrifying dark elegance, than to sort pebbles into prime-numbered heaps.  A galaxy transformed into pebbles and sorted into prime-numbered heaps would be just plain boring.

The Pebblesorters, of course, would only reason that music is p-pointless because it doesn't help you sort pebbles into heaps; the human activity of humor is not only p-pointless but just plain p-bizarre and p-incomprehensible; and most of all, the human vision of a galaxy in which agents are running around experiencing positive reinforcement but not sorting any pebbles, is a vision of an utterly p-arbitrary galaxy devoid of p-purpose.  The Pebblesorters would gladly sacrifice their lives to create a P-Friendly AI that sorted the galaxy on their behalf; it would be the most p-profound statement they could make about the p-meaning of their lives.

So which of these two perspectives do I choose?  The human one, of course; not because it is the human one, but because it is right.  I do not know perfectly what is right, but neither can I plead entire ignorance.

And the Pebblesorters, who simply are not built to do what is right, choose the Pebblesorting perspective: not merely because it is theirs, or because they think they can get away with being p-arbitrary, but because that is what is p-right.

And in fact, both we and the Pebblesorters can agree on all these points.  We can agree that sorting pebbles into prime-numbered heaps is arbitrary and unjustified, but not p-arbitrary or p-unjustified; that it is the sort of thing an agent p-should do, but not the sort of thing an agent should do.

I fully expect that even if there is other life in the universe only a few trillions of lightyears away (I don't think it's local, or we would have seen it by now), that we humans are the only creatures for a long long way indeed who are built to do what is right.  That may be a moral miracle, but it is not a causal miracle.

There may be some other evolved races, a sizable fraction perhaps, maybe even a majority, who do some right things.  Our executing adaptation of compassion is not so far removed from the game theory that gave it birth; it might be a common adaptation.  But laughter, I suspect, may be rarer by far than mercy.  What would a galactic civilization be like, if it had sympathy, but never a moment of humor?  A little more boring, perhaps, by our standards.

This humanity that we find ourselves in, is a great gift.  It may not be a great p-gift, but who cares about p-gifts?

So I really must deny the charges of moral relativism:  I don't think that human morality is arbitrary at all, and I would expect any logically omniscient reasoner to agree with me on that.  We are better than the Pebblesorters, because we care about sentient lives, and the Pebblesorters don't.  Just as the Pebblesorters are p-better than us, because they care about pebble heaps, and we don't.  Human morality is p-arbitrary, but who cares?  P-arbitrariness is arbitrary.

You've just got to avoid thinking that the words "better" and "p-better", or "moral" and "p-moral", are  talking about the same thing—because then you might think that the Pebblesorters were coming to different conclusions than us about the same thing—and then you might be tempted to think that our own morals were arbitrary.  Which, of course, they're not.

Yes, I really truly do believe that humanity is better than the Pebblesorters!  I am not being sarcastic, I really do believe that.  I am not playing games by redefining "good" or "arbitrary", I think I mean the same thing by those terms as everyone else.  When you understand that I am genuinely sincere about that, you will understand my metaethics.  I really don't consider myself a moral relativist—not even in the slightest!

 

Part of The Metaethics Sequence

Next post: "You Provably Can't Trust Yourself"

Previous post: "Is Fairness Arbitrary?"

FairnessEthics & Morality
Personal Blog

25

Mentioned in
The Bedrock of Morality: Arbitrary?
New Comment
119 comments, sorted by
oldest
Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]Hopefully_Anonymous00

I'm fine with a galaxy without humor, music, or art. I'd sacrifice all of that reproductive fitness signalling (or whatever it is) to maximize my persistence odds as a subjective conscious entity, if that "dilemma" was presented to me.

Reply
1diegocaleiro
This is because you are more strongly (individually, not the whole of our species) to maximize your survival opportunities than your pleasure opportuinities. If it is the case, for instance, that you would accept to become an entity with a constant focus of attention into almost nothing (a meditating entity) instead of having loads of fun for one fourth of the time (say, 1000 versus 4000 years) this means you are a survival optimization process. You were not designed for this, you were designed to maximize genetic fitness. And you were designed to maximize memetic fitness. You have been designed by two major forces composed of many many minor forces. You have decided, morally, to abdicate all this plurality in the name of conscious survival. You praise individuality, and you praise persistence. I would suggest to let one more meme take over your mind. The meme of fun. Optimize for fun, don't lead us into a universe of "sperm donors" who detect minimal entropy spaces to be at for longer. Don't lead us here: http://www.nickbostrom.com/fut/evolution.html
0VAuroch
I would not, or at least probably would not. For sufficiently drastic survival ratio, I might. I would not, because humor and art are tied far too directly into the sense of fun; an excellent game is art just as much as Guernica, Citizen Kane, or a Beethoven symphony, and losing that entire section of the aesthetic sense would spoil too many fun things. Of course, whatever senses replaced our senses of art/humor might bring other, equal fun with them. In which case I'd be more inclined to accept the deal.
[-]Nominull360

I kind of think humor, music and art are pretty cool, myself.

Reply
[-]Psy-Kosh20

Hopefully: But would you replace those with anything else? I'd want persistance, but I'd want growth and, well, fun! :)

Reply
[-]roko370

I think that your use of the word arbitrary differs from mine. My mind labels statements such as "we should preserve human laughter for ever and ever" with the "roko-arbitrary" label. Not that I don't enjoy laughter, but there are plenty of things that I presently enjoy that, if I had the choice, I would modify myself to enjoy less. Activities such as enjoying making fun of other people, eating sweet foods, etc. It strikes me that the dividing line between "things I like but wish I didn't like" and "things I like and want... (read more)

Reply
[-]Z._M._Davis50

Eliezer: "I am not playing games by redefining 'good' or 'arbitrary' [...]"

I imagine the counterargument would be that while you're not playing e-games by e-redefining the terms, you are playing games by redefining the terms.

Upon a preview it looks like Roko beat me to it.

Reply
[-]roko380

It also worries me quite a lot that eliezer's post is entirely symmetric under the action of replacing his chosen notions with the pebble-sorter's notions. This property qualifies as "moral relativism" in my book, though there is no point in arguing about the meanings of words.

My posts on universal instrumental values are not symmetric under replacing UIVs with some other set of goals that an agent might have. UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X. Maybe I find this satisfying because I have always been more at home with category theory than logic; I have defined a set of values by requiring them to satisfy a universal property.

Reply
[-]Seinberg10

But laughter, I suspect, may be rarer by far than mercy.

Curious why you suspect this. Is it particularly mammalian in some respect? I confess I could be naive, but it seems to me that any sufficiently intelligent being/agent would be just as likely as we humans are to have humor. I suppose that raises the question of how likely it is, and are we just incredibly lucky to have inherited such a trait. Still, it's such a core aspect of so much of our species -- even more than mercy, I think! -- that I'm curious why you think that.

Reply
0VAuroch
Only humans laugh, and only the chimpanzee family does anything similar; even that isn't like normal laughter, but the kind of laughter you express when tickled (which is subjectively qualitatively different from laughter at a joke).
[-]J_Thomas230

Eliezer, you claim that there is no necessity we should accept Dennis's claim he should get the whole pie as fair. I agree.

There is also no necessity he should accept our alternative claim as fair.

There is no abstract notion that is inherently fair. What there is, is that when people do reach agreement that something is fair, then they have a little bit more of a society. And when they can't agree about what's fair they have a little less of a society. There is nothing that says ahead of time that they must have that society. There is nothing that says ahe... (read more)

Reply
[-]Richard_Hollerith240

Though Eliezer does not say it explicitly today, the totality of his public pronouncements on laughter leads me to believe that he considers laughter an instrinsic good of very high order. I hope he does not expect me to accept the highness of the probability of the rareness of humor in the universe as evidence for humor's intrinsic goodness. After all, spines are probably very rare in the universe, too. At least spines with 32 (or however many humans have) vertebrae are.

Eliezer does not explicitly say today that happiness is an intrinsic good, but he d... (read more)

Reply
[-]Kip_Werking40

I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary.

Reply
[-]Wiseman10

@J Thomas: "Why would anybody think that there is a single perfect morality, and if everybody could only see it then we'd all live in peace and harmony?"

Because they have a specific argument which leads them to believe that?

You know, there's no reason why one couldn't consider one language more efficient at communication than others, at least by human benchmarks, all else being equal (how well people know the language, etc.). Ditto for morality.

Thomas, you are running in to the same problem Eliezer is: you can't have a convincing argument about w... (read more)

Reply
[-]Richard_Hollerith200

Eliezer can reply that moral conclusions are different, so the sermon does not apply. Well, I think it should apply, in certain case, such as when you are contemplating the launch of the seed of a superintelligence, which is an occasion that IMO demands a complete reevaluation of one's terminal values and the terminal values of one's society.

Reply
[-]Eliezer Yudkowsky30
I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary.

...

Reply
[-]Richard_Hollerith200

Kip and Wiseman slipped in. Busy tonight.

Reply
[-]Z._M._Davis50

Richard: "Eliezer answer might refer to the difference between the simplicity of Autobliss 1.0 and the complexity of a human."

I'm pretty sure he wouldn't say that. Rather, the claim (if I'm reading him correctly) is that the true referent of good really is a really complicated bundle of human values. In a material universe, you can't cash out "intrinsic goodness" in the intuitive way.

Reply
[-]komponisto2100

I'm really having trouble understanding how this isn't tantamount to moral relativism -- or indeed moral nihilism. The whole point of "morality" is that it's supposed to provide a way of arbitrating between beings, or groups, with different interests -- such as ourselves and Pebblesorters. Once you give up on that idea, you're reduced, as in this post, to the tribalist position of arguing that we humans should pursue our own interests, and the Pebblesorters be damned. When a conflict arises (as it inevitably will), the winner will then be whoever has the bigger guns, or builds AI first.

Mind you, I don't disagree that this is the situation in which we in fact find ourselves. But we should be honest about the implications. The concept of "morality" is entirely population-specific: when groups of individuals with common interests come into contact, "morality" is the label they give to their common interests. So for us humans, "morality" is art, music, science, compassion, etc. in short, all the things that we humans (as opposed to Pebblesorters) like. This is what I understand Eliezer to be arguing. But if this is your position, you may as well ... (read more)

Reply
3Yosarian2
I don't think so. I don't think it's h-right to destroy or reprogram the pebblesorters, so if we're exploring space and find them, I don't think we'll do that. It may be P-right for them to re-program us to forget about h-right and just start sorting pebbles, though, so we want to watch out to make sure that doesn't happen.. I think the mistake that most "moral relativists" make is that they forget about the shared human morality we all have, and therefore claim that it's all arbitrary and meaningless.
-2MugaSofer
Could you taboo "point" and "arbitrating"? I'm not sure if I'm interpreting this correctly.
0A1987dM
He's also arguing that the pebblesorters p-shold pursue their own p-interests and we humans be p-damned, for that matter.
0TheAncientGeek
Moral realists don't believe in karmic retribution. most popular critiques of MR strawman it as something much stronger than any other kind of realism or objectivism. Objectivisms only require the availability of mind independent truths to those capable of, and interested in finding them.
[-]Carl_Shulman00

Roko,

"UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X." Roko,

You know that all of the so-called 'UIVs' that have been postulated only apply for some Y under some conditions (the presence of other powerful agents and game theoretic considerations or manipulation, self-referential utility functions, preferences over mathematical truths, and many other considerations make so-called UIVs useless or sources of terminal disvalue for an infinite number of cases), and an agent could have the ter... (read more)

Reply
[-]MichaelAnissimov00

Kip, can you expound?

Reply
[-]Tarzan,_me_Jane2-10

There has never been, so far as I able to determine, any force so unfriendly to humans as humans. Yet we read day after day about one very smart man's philosophizing about the essence of humanity, supposedly so that it can be included in the essence of fAI. Wouldn't it be incredible if tomorrow, or sometime in the near future, someone who has been working and actually come up with some designs for fAI or AGI produces a real product, and it makes all the hubris of these responses irrelevant? What is the purpose of an intelligence that is able to take all t... (read more)

Reply
[-]komponisto200

Clarification: in the first paragraph of the above comment, when I wrote "The whole point of 'morality' is..." what I meant was "The whole point of non-relativist 'morality' is...".

Reply
[-]Carl_Shulman00

"I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary."

Kip,

Given Eliezer's definition of rightness (which is different from current object-level views), if there is a sufficiently cogent and convincing argument for pebblesorting, then pebblesorting is both right and p-right. Do you think that there is a significant chance you would ever view pebblesorting as k-right with expanded intelligence and study?

Reply
[-]Vladimir_Nesov00

It looks like fairness can be said to be f-morality, built from current morality so that it is known to be sufficiently stable under reflection (that is, (meta)*-f-moral), and as moral as possible. While we travel the road of moral progress, avoiding getting trapped in the simplistic ditches of fake moralities, we need a solid target for agreement, and this is what a particular fairness is. Morality unfolds in a moral way, while casting a shadow of unfolded fairness.

Reply
[-]Richard_Hollerith200

Yes, Z.M., human happiness is not what Eliezer plans to use the superintelligence to maximize. Good to make that clear. But it might be worthwhile to question the intrinsic goodness of human happiness, as a warm-up to questioning the coherent extrapolated volition (CEV) of the humans.

Reply
[-]J_Thomas200

But most of all - why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good? What is even the appeal of this, morally or otherwise? At all?

I don't think you ought to try to optimise fitness. Your opinion about fitness might be quite wrong, even if you accept the goal of optimising fitness. Say you sacrifice trying to optimise fitness and then it turns out you failed. Like, you try to optimise for intelligence just before a plague hits that kills 3/4 of the public. You should have optimised for... (read more)

Reply
[-]J_Thomas200

"Why would anybody think that there is a single perfect morality, and if everybody could only see it then we'd all live in peace and harmony?"

Because they have a specific argument which leads them to believe that?

Sure, but have you ever seen such an argument that wasn't obviously fallacious? I have not seen one yet. It's been utterly obvious every time.

Thomas, you are running in to the same problem Eliezer is: you can't have a convincing argument about what is fair, versus what is not fair, if you don't explicitly define "fair" in the f... (read more)

Reply
[-]Tim_Tyler00

Re: why on Earth would any human being think that one ought to optimize inclusive genetic fitness

"Ought" is a word that only makes sense in the context of an existing optimisation strategy. As far as biologists can reasonably tell, the optimisation strategy of organisms involves maximising their inclusive genetic fitness. So the short answer to this is: because nature built them that way.

The bigger puzzle is not why organisms act to maximise their inclusive genetic fitness, but why they sometimes systematically fail to do so. What cognitive ma... (read more)

Reply
[-]Tim_Tyler10

For evolution being wasteful, see: http://alife.co.uk/essays/evolution_is_good/

For evolution being stupid, see: http://alife.co.uk/essays/evolution_sees/

Reply
[-]Virge200

komponisto: "I'm really having trouble understanding how this isn't tantamount to moral relativism"

I think I see an element of confusion here in the definition of moral relativism. A moral relativist holds that "no universal standard exists by which to assess an ethical proposition's truth". However, the word universal in this context (moral philosophy) is only expected to apply to all possible humans, not all conceivable intelligent beings. (Of all the famous moral relativist philosophers, how many have addressed the morals of general ... (read more)

Reply
[-]Eliezer Yudkowsky50

You can apply the standard of goodness to all intelligent beings, no problem. It's just that they won't apply it to themselves.

The content of "good" is an abstracted idealized dynamic, or as Steven put it, a rigid designator (albeit a self-modifying rigid designator). Thus what is good, or what is not good, is potentially as objective as whether a pile of pebbles is prime. It is just that not every possible optimization process, or every possible mind, does what is good. That's all.

Reply
[-]Ben_Jones10

I wish Arnie's character had made a long speech in the middle of the film explaining that Predator wasn't evil or even wrong, he was just working around a different optimization process.

@HA

I'm fine with a galaxy without humor, music, or art. I'd sacrifice all of that...to maximize my persistence odds as a subjective conscious entity....

So existing is a terminal value in and of itself for you HA. Wouldn't you get bored? Or would you try to excise your boredom circuits, along with your humour, music and art circuits? How about your compassion circuits? Do yo... (read more)

Reply
[-]Ian_C.00

In the real world, everything worth having comes from someone's effort -- even wild fruit has to be picked, sorted, and cleaned and fish need to be caught, gutted etc. I think this universal fact of required effort is probably part of the data we get the concept of fairness from in the first place, so reasoning in a space where pies pop in to existence from nothing seems like whatever you conclude might not be applicable to the real world anyway.

Reply
[-]Hopefully_Anonymous00

Ben, you write "Do you strive for the condition of perfect, empty, value-less ghost in the machine, just for its own sake...?".

But my previous post clearly answered that question: "I'd sacrifice all of that reproductive fitness signalling (or whatever it is) to maximize my persistence odds as a subjective conscious entity, if that "dilemma" was presented to me."

Reply
[-]Mario200

It is pretty clever to suggest objective morality without specifying an actual moral code, as it is always the specifics that cause problems.

My issue would be how Eliezer appears to suggest that human morality and alien morality could be judged separately from the genetics of each. Would super intelligent alien bees have the same notions of fairness as we do, and could we simply transplant our morality onto them, annd judge them accordingly, with no adjustments made for biological differences? I think it is very likely that such a species would consider ... (read more)

Reply
[-]Lakshmi10

To be fair (cough), your argument that '5 people means the pie should be divided into 5 equal parts' assumes several things...

1) Each person, by virtue of merely being there, is entitled to pie.

2) Each person, by virtue of merely being there, is entitled to the same amount of pie as every other person.

While this division of the pie may be preferable for the health of the collective psyche, it is still a completely arbitrary (cough) way to divide the pie. There are several other meaningful, rational, logical ways to divide the pie. (I believe I suggested on... (read more)

Reply
[-]Roko10

Eliezer: "I really don't consider myself a moral relativist - not even in the slightest!"

Meta-ethical relativism (wikipedia)

Meta-ethical relativists, in general, believe that the descriptive properties of terms such as "good", "bad", "right", and "wrong" do not stand subject to universal truth conditions, but only to societal convention and personal preference. Given the same set of verifiable facts, some societies or individuals will have a fundamental disagreement about what one ought to do based on socie... (read more)

Reply
[-]J_Thomas200

Lakshmi, Eliezer does have a point, though.

While there are many competing moral justifications for different ways to divide the pie, and while a moral relativist can say that no one of them is objectively correct, still many human beings will choose one. Not even a moral relativist is obligated to refrain from choosing moral standards. Indeed, someone who is intensely aware that he has chosen his standards may feel much more intensely that they are his than someone who believes they are a moral absolute that all honest and intelligent people are obligated ... (read more)

Reply
[-]Caledonian220

As far up as you go, there's no level that calls for unconditional surrender.
I see no reason to presume that a concept of fairness would never require that one involved entity cede their demands and give in.

Reply
[-]Larry_D'Anna-10

Roko: What the heck does morality have to do with category theory at all?

Reply
[-]Eliezer Yudkowsky70
Meta-ethical relativists, in general, believe that the descriptive properties of terms such as "good", "bad", "right", and "wrong" do not stand subject to universal truth conditions, but only to societal convention and personal preference. Given the same set of verifiable facts, some societies or individuals will have a fundamental disagreement about what one ought to do based on societal or individual norms, and one cannot adjudicate these using some independent standard of evaluation. The latter standard will alway
... (read more)
Reply
1lmm
If you follow through on this view it seems to lead to the position that everyone has their own referent for "good", and there is no meaningful way for two different humans to argue about whether a given action is good. Which would suggest there is little point trying to persuade other people to be good, or hoping to collaboratively construct a friendly AI (since an l-friendly AI is unlikely to be e-friendly).
4wedrifid
Cooperation does not require modification of others to have identical values. Even agents with actively opposed values can cooperate (and so create a mutually friendly AI) so long as the opposition is not perfect in all regards.
3lmm
This site has been at pains to emphasise that an AI will be an optimization process of never-before-seen power, rewriting reality in ways that we couldn't possibly predict, and as such an AI whose values are even slightly misaligned with one's own would be catastrophic for one's actual values.
4wedrifid
What is relevant to the decision to create or prevent such an AI from operating is the comparison between what will occur in the absence of the AI and what the AI will do. For example gwern's values are not identical to mine but if I had the choice between pressing a button to release an FAI or a button to destroy it then I would press the button to release it. FAI isn't as good as FAI (by subjective tautology) but FAI is overwhelmingly better than nothing. I expect FAI to allow me to live for millions of years, and for the cosmic commons to be exploited to do things that I generally approve of. Without that AI I think it is most likely that myself and my species will go to oblivion. The above doesn't even take into account cooperation mechanisms. That's just flat acceptance of optimisation for another's values over distinctly sub-optimisation of my own. When it comes to agents with conflicting values cooperating negotiation applies and if both agents are rational and in a situation where mutual FAI creation is possible but unilateral FAI creation can be prevented then the result will be an FAI that optimises for a compromise of the value systems. To whatever extent the values of the two agents are not perfectly opposed this outcome will be superior to the non-cooperative outcome. For example if gwern and I were in such a situation the expected result would be the release of FAI>. Neither of us will prefer that option over the FAI that is personalised to ourselves but there is still a powerful incentive to cooperate. That outcome is better than what we would have without cooperation. The same applies if a paperclip maximiser and a staple maximiser are put in that situation. (It does not apply is a paperclip maximiser meets a paperclip minimiser.)
[-]Roko20

@Eliezer: "what one ought to do" vs. "what one p-ought to do"

Suppose that the pebblesorter civilization and the human civilization meet, and (fairly predictably) engage in a violent and bitter war for control of the galaxy. Why can you not resolve this war by bringing the pebblesorters and the humans to a negotiating table and telling them "humans do what they ought to do and Pebblesorters do what they p-ought to do"?

You cannot play this trick because p-ought is grounded in what the pebblesorters actually do, which is in turn ... (read more)

Reply
[-]Eliezer Yudkowsky50
You cannot play this trick because p-ought is grounded in what the pebblesorters actually do, which is in turn grounded in the state of the universe they aim for, which is the same universe that we live in. The humans and the pebblesorters seem to be disagreeing about something as they fight each other

Does a human being disagree with natural selection? About what, exactly? How would we argue natural selection into agreement with us?

Standard game theory talks about interactions between agents with different goals. It does not presume that all agents mu... (read more)

Reply
2A1987dM
It so happens that Pebblesorters are the sort of beings who do p-good, and that humans are not; but this is a mere happenstance of a p-moral miracle, not the p-justification for sorting pebbles instead of having fun. :-)
[-]Tim_Tyler20

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox's stomach. They may argue passionately about the rabbit's fate - and even stoop to violence.

Reply
[-]Eliezer Yudkowsky40

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox's stomach. They may argue passionately about the rabbit's fate - and even stoop to violence.

Really? I would be interested in hearing their philosophical arguments then as for why the rabbit should be eating grass or the rabbit should be in the fox's stomach. I understand, of course, that the rabbit does eat grass and that the fox does hunt the rabbit, but I was not aware ... (read more)

Reply
[-]Roko10

Eliezer: No, "good" is defined as that which leads to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc.

  • you're going to have to give me a definition that doesn't involve "etc" for me to critique this further. As I understand it, you've taken some averaged or CEV'd collection of things that humans like doing, or thing
... (read more)
Reply
[-]Tim_Tyler10
I would be interested in hearing their philosophical arguments then as for why the rabbit should be eating grass or the rabbit should be in the fox's stomach. I understand, of course, that the rabbit does eat grass and that the fox does hunt the rabbit, but I was not aware that these were persuasive moral arguments.

They are to the parties in question:

The rabbit argues that if it is eaten by the fox, then it will die - and that should not happen.

The fox argues that if it doesn't eat rabbits, then it will die - and that should not happen.

Neither considers ... (read more)

Reply
[-]Roko00

@Eli: All attempts to justify an ethical theory take place against a background of what-constitutes-justification. You, for example, seem to think that calling something "universally instrumental" constitutes a justification for it being a terminal value, whereas for me this is a nonstarter. For every mind that thinks that terminal value Y follows from moral argument X, there will be an equal and opposite mind who thinks that terminal value not-Y follows from moral argument X. I do indeed have a word for theories that deny this: I call them "... (read more)

Reply
[-]Allan_Crossman30

Eliezer, I think I kind-of understand by now why you don't call yourself a relativist. Would you say that it's the "psychological unity of mankind" that distinguishes you from relativists?

A relativist would stress that humans in different cultures all have different - though perhaps related - ideas about "good" and "right" and so on. I believe your position is that the bulk of human minds are similar enough that they would arrive at the same conclusions given enough time and access to enough facts; and therefore, that it's an ... (read more)

Reply
[-]Tim_Tyler10
It would seem that you are not distinguishing between what a system does and what it should do.

In my book, there's not really any such thing as what a system should do.

Should only makes sense with respect to the morals of some agent.

If you don't specify an agent, should becomes an extremely vague and ambiguous term.

Should statements are not about what happens, but about the desirability of what might happen - according to the moral system of some agent.

Reply
[-]Z._M._Davis30

Concerning the charge of relativism: it seems clear that Eliezer is a moral relativist in the way that the term is normally understood, but not as he understands it. There may be a legitimate dispute here, but as far as communication goes, we should not be having problems. In deference to common usage, I would reserve right for the moral realism of Roko et al. and use something like h-right for Eliezer's notion of humanity's abstracted idealized dynamic--but I don't think it really matters right now.

Roko writes: "My list is the current human notion of... (read more)

Reply
[-]Jadagul20

Eliezer, I think you come closer to sharing my understanding of morality than anyone else I've ever met. Places where I disagree with you:

First, as a purely communicative matter, I think you'd be clearer if you replaced all instances of "right" and "good" with "E-right" and "E-good."

Second, as I commented a couple threads back, I think you grossly overestimate the psychological unity of humankind. Thus I think that, say, E-right is not at all the same as J-right (although they're much more similar than either is to... (read more)

Reply
[-]Caledonian240

Odd how, despite the psychological unity of mankind and the ease of 'extrapolating' human volition, these discussions always seem to end in establishing specialized words to refer to the perceptions and beliefs of specific individuals.

Reply
[-]Yvain250

Why "ought" vs. "p-ought" instead of "h-ought" vs. "p-ought"?

Sure, it might just be terminology. But change

"So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is right."

to

"So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is h-right."

and the difference between "because it is the human one" and "because it is h-right" sounds a lot less convincing.

Reply
0MarsColony_in10years
If I see a toddler in the path of a boulder rolling downhill, I don't ask myself "should I help the bolder, or the toddler?" and conclude "the toddler, because it is the human one." If I were to even pause and ask myself a question, it would be "what should I do?" and the answer would be "save the toddler, because it is h-right". Perhaps h-right is "just the human perspective", but that's not the reason I save the toddler. Similarly, the bolder rolls downhill because F=G(m1m2)/r^2, not because it is what boulders do. It is what boulders do, but that's different from the question of why they do what they do.
[-]Nominull280

To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

Reply
0[anonymous]
This made me laugh, upvoted.
[-]J_Thomas200

Nominull, don't the primalists have a morality about heaps of stones?

They believe there are right ways and wrong ways to do it. They sometimes disagree about the details of which ways are right and they punish each other for doing it wrong.

How is that different from morality?

Reply
[-]Nominull220

If you've ever taken a mathematics course in school, you yourself may have been introduced to a situation where it was believed that there were right and wrong ways to factor a number into primes. Unless you were an exceptionally good student, you may have disagreed with your teacher over the details of which way was right, and been punished for doing it wrong.


It strikes me as plainly apparent that math homework is not morality.

Reply
[-]Eliezer Yudkowsky90
To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

Thank you, Nominull. I'm glad someone gets it, anyway.

Reply
[-]J_Thomas200

If you've ever taken a mathematics course in school, you yourself may have been introduced to a situation where it was believed that there were right and wrong ways to factor a number into primes. Unless you were an exceptionally good student, you may have disagreed with your teacher over the details of which way was right, and been punished for doing it wrong.

My experience with math classes was much different from yours. When we had a disagreement, the teacher said, "How would we tell who's right? Do you have a proof? Do you have a counter-example?&q... (read more)

Reply
[-]Yvain2170

To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

But by Eliezer's standards, it's impossible for anyone to be a relativist about anything.

Consider what Einstein means when he says time and space are relative. He doesn't mean you can just say whatever you want about them, he means that they're relative to a certain reference frame. An observer on Earth may think it's five years since a spaceship launched, and an observer on the spaceship may think it's only been one, and each of them is correct relative to their reference frame.

We could define "time" to mean "time as it passes on Earth, where the majority of humans live." Then an observer on Earth is objectively correct to believe that five years have passed since the launch. An observer on the spaceship who said "One year has passed" would be wrong; he'd really mean "One s-year has passed." Then we could say time and space weren't really relative at all, and people on the groun... (read more)

Reply
0A1987dM
Until I read that, I thought I understood (and agreed with) Eliezer's point, but that got me thinking. Now, I guess Eliezer would agree that it's easy for Japanese people to speak Japanese, while he wouldn't agree that it's right for Baby-Eaters to keep on eating their children. So there must be something subtler I'm missing.
1TheOtherDave
FWIW, my understanding of the original claim was precisely that morality is special in this way: that it means something to describe what humans value as "right" compared to what nonhumans value (and what nobody values), whereas it doesn't mean anything analogous to describe the languages humans speak as "easily speakable" compared to the languages nonhumans speak (and the languages nobody speaks). And whatever that something is, eating babies simply doesn't possess it, even for Baby-Eaters. Personally I've never understood what that something might be, though, nor seen any evidence that it exists.
0nshepperd
Have you forgotten that what it means to describe something by a word is given precisely by the sense of that word that the speaker has in mind? That you can call eudaimonia "right", and heaps of prime pebbles "prime" is a fact about the words "right" and "prime" as used by humans, not about eudaimonia and pebbles themselves (except insofar as eudaimonia and prime-pebbled heaps by their nature satisfy the relevant definitions of "right" and "prime", of course). Is English the easiest language, if you define "easiest" as "easiest for an English-speaker to speak"? How many legs does a dog have, if you call a tail a leg?
0TheOtherDave
When I assert "eudaimonia is right" (supposing I believed that), there exist two structures in my brain, S1 and S2, such that S1 is tagged with the lexical entry "right" and S2 is tagged with the lexical entry "eudaimonia", and S1 and S2 are related such that if my brain treats some thing X as an instance of S2, it also treats X as having the property S1. Well, for a certain use of "is," anyway.
0nshepperd
I was going to ask how that relation came about, and how it behaves when your brain is computing counterfactuals... but even though those are good questions to consider, I realised that wouldn't really be that helpful. So... What I'm really trying to say is that there's nothing special about morality at all. There doesn't have to be anything special about it for eudaimonia to be right and for pebble-sorting to not. It's just a concept, like every other concept. One that includes eudaimonia and excludes murder, and is mostly indifferent to pebble-sorting. Same as the concept prime includes {2, 3, 5, 7, ...} and excludes {4, 6, 8, 9, ...}. The only thing remotely "special" about it is that it happens to be a human terminal value -- which is the only reason we care enough to talk about it in the first place. The only thing remotely special about the word "right" is that it happens to mean, in english, this morality-concept (which happens to be a human terminal value). So, to say that "eudaimonia is right" is simply to assert that eudaimonia is included in this set of things that includes eudaimonia and excludes murder (in other words, yes, "X ∈ S2 implies X ∈ S1", where S2 is eudaimonia and S1 is morality). To say that what babyeaters value is right would be to assert that eating babies is included in this set ("X ∈ babyeating implies X ∈ S1", which is clearly wrong, since murder ∈ babyeating).
0Ghatanathoah
I generally agree with everything you say here, except that I'd like to clarify what you mean by "special" when you say that morality need not be special, as I'm not sure it would be clear to everyone reading your post. Obviously morality has no mystical properties or anything. It isn't special in that sense, which is what I think you mean. But morality does differ (in a totally nonmystical way) from many other terminal values in being what Eliezer calls "subjectively objective and subjunctively objective." That is, there is only one way, or at least an extremely limited number of ways, to do morality correctly. Morality is not like taste, it isn't different for every person. You obviously already know this, but I think that it's important to make that point clear because this subject has huge inferential distances. Hooray for motivational externalism!
0TheOtherDave
Yeah, it's precisely the assumption that the computation we refer to by "morality" is identical for every human that makes this whole approach feel inadequate to me. It's just not clear to me that this is true, and if it turns out not to be true, then we're faced with the problem of reconciling multiple equally valid moralities. Of course, one approach is to stop caring about humans in general, and only care about that subset of humanity that agrees with me.
0nshepperd
You mean, the assumption that every human uses the word "morality" to refer to the same computation. Clearly, if I use "morality" to refer to X, and you also use the word "morality" to refer to X, then X and X are identical trivially. We refer to the same thing. Keep careful track of the distinction between quotation and referent. Anyway, before I answer, consider this... If other people use "morality" to refer to something else... then what? How could it matter how other people use words?
2TheOtherDave
I agree that if you and I both use "morality" to refer to X, then we refer to the same thing. If I use "morality" to refer to X1 and you use it to refer to X2, it doesn't matter at all, unless we try to have a conversation about morality. Then it can get awfully confusing. Similar things are true if I use "rubber" to refer to a device for removing pencil marks from paper, and you use "rubber" to refer to a prophylactic device... it's not a problem at all, unless I ask you to fetch a bunch of rubbers from the supply cabinet for an upcoming meeting.
0nshepperd
But what does that mean for what you should do? Nothing, right? It doesn't matter that someone else uses "morality" to refer to X2. If I call murder "right", murder is still wrong. And you should still lock up murderers.
0TheOtherDave
Talking about the truth-value of the assertion "murder is right" seems unjustified at this point, much like the truth-value of "rubbers help prevent pregnancy." Is it true? Yes. Is it false? Yes. When a word means different things within a conversation, ambiguity is introduced to many sentences containing that word. It helps at that point to set aside the ambiguous label and introduce more precise ones. Which is why I introduced X1 and X2 in the first place. I agree that the fact that X1 rejects murder doesn't necessarily change just because X2 endorses it. But I don't agree that what X1 endorses is necessarily independent of what X2 endorses. For example, if I don't value the existence of Gorgonzola in the world, and I value your preferences being satisfied, then I value Gorgonzola IFF you prefer there exist Gorgonzola in the world. To the extent that what I should do is a function of what I value, and to the extent that X2 relates to your preferences, then X2 (what you call "right") has a lot to do with what I should do.
2nshepperd
The assertion "murder is right" -- by your definition of "right", which is the only definition you should care about, being the person who formulates the question "what is right for me to do?" -- has a value of TRUE precisely if X1 endorses murder. There's nothing unjustified about saying that, since X1 was brought in specifically defined as the thing your definition of "right" refers to. I'll grant that it's perfectly possible that X1 might have a term in it (to borrow terminology from the utility function world) for other peoples' terminal values. But if so that's a question of object-level ethics, not meta-ethics.
1TheOtherDave
It is not clear to me that X1 is the only definition of "right" I should care about, even if it is mine... any more than thing-to-erase-pencil-marks-with is the only definition of "rubber" I should care about. Regardless, whether I should care about other people's definitions of these words or not, the fact remains that I do seem to care about it. And I also seem to care about other people's preferences being satisfied, especially the preferences that they associate with the emotional responses that lead them to talk about that preference being "right" (rather than just "my preference"). Again, maybe I oughtn't... though if so, it's not clear to me why... but nevertheless I do. It may be relevant that this is not the only moral question I formulate. Other moral questions include "what is right for others to do?" and "what is right to occur?" Indeed, that last one is far more important to me than the others, which is one reason I consider myself mostly a consequentialist. Maybe so. What follows from that?
3nshepperd
Any question you could possibly want the answer to relating in any sense to "rightness" is not a question at all unless you have a definition of "right" in mind (or at the least a fuzzy intuitive definition that you don't have full access to). You want to know "what is right to occur". You won't get anywhere unless you have an inkling of what you meant by "right". It's built into the question that you are looking for the answer to your question. It's your question! Maybe you decide that X1 (which is the meaning of your definition of "right") includes, among things such as "eudaimonia" and "no murder", "other humans getting what they value". Then the answer to your question is that it's right for people to experience eudaimonia and to not be murdered, and to get what they value. And the answer to "what should I do" is that you should try and bring those things about.
0TheOtherDave
Yes, that's true. Or maybe I decide that X1 doesn't include other humans getting what they value, and I'm only under the impression that it does because there are some things that other humans happen to value that X1 does include, or because X1 includes something that is similar but not quite identical to other humans getting what they value, or for some other reason. Either way, whichever of those things turns out to be the case, that's what I should do... agreed (1). Of course, in some of those cases (though not others), in order to work out what that is in practice, I also need to know what other humans' equivalents of X1 are. That is, if it turns out X1 includes you getting what you value as long as you're alive, and what you value is given by X2, then as long as you're alive I should bring about X2 as well as X1. And in this scenario, when you are no longer alive, I no longer should bring about X2. ==== (1) Or, well, colloquially true, anyway. I should certainly prefer those things occurring, but whether I should do anything in particular, let alone try to do anything in particular, is less clear. For example, if there exists a particularly perverse agent A who is much more powerful than I, and if A is such that A will bring about what I value IFF I make no efforts whatsoever towards bringing them about myself, then it follows that what I ought to do is make no efforts whatsoever towards bringing them about. It's not clear that I'm capable of that, but whether I'm capable of it or not it seems clear that it's what I ought to do. Put a different way, in that situation I should prefer to be capable of doing so, if it turns out that I'm not.
0Ghatanathoah
As I said before: If morality refers to a large computation related to the wellbeing of eudaemonic creatures it might be possible that some people value different aspects of wellbeing more than others (i.e. some people might care more about freedom, others more about harm). But there'd still be a huge amount of agreement. I think a good analogy is with the concept of "health." It's possible for people to care about different aspects of health more. Some people might care more about nutrition, others about exercise. But there are very few ways to be healthy correctly, and near infinite ways to be unhealthy. And even if someone thinks you have your priorities wrong when trying to be healthy, they can still agree that your efforts are making you healthier than no effort at all. I care about the wellbeing of animals to some extent, even though most of them don't care about morality at all. I also care, to a limited extent, about the wellbeing of sociopathic humans even though they don't care about morality at all. I admit that I don't care about them as much as I do about moral beings, but I do care. If other moral humans have slightly different moral priorities from you I think they'd still be worth a great deal of caring. Especially if you care at all about animals or sociopaths, who are certainly far less worthy of consideration than people who merely disagree with you about some aspect of morality.
0TheOtherDave
I agree that we should expect significant (though not complete) overlap within the set of moral judgments made by all humans. I would expect even more overlap among those made by non-pathological humans, and even more overlap among those made by non-pathological humans who share a cultural heritage. I would expect less overlap (though not zero) among the set of moral judgments made by non-humans. I agree that if statement X (e.g. "murder is wrong") is endorsed by all the moral judgments in a particular set, then the agents making those judgments will all agree that X is right, although perhaps to different degrees depending on peripheral particulars. Similarly, if statement Y is not endorsed by all the moral judgments in a particular set, then the agents making those judgments will not all agree that X is right. It's clear in the first case that right action is to abide by the implications of X. In the second case, it's less clear what right action is.
1Ghatanathoah
I think the point I am trying to get across, and one of the major points made by Eliezer in this sequence is that some of the things you are referring to as moral judgements aren't really moral judgements. Eliezer is basically saying that when you make a moral judgement you are making computations about various aspects of the wellbeing of eudaemonic creatures. A judgement that refers to the huge and complex concept "the wellbeing of eudaemonic creatures" is a moral judgement. A judgement that refers to some other concept is not a moral judgement, even if we use the same word to describe each. When a sociopath says "It is good for me to kill people" he is not making a moral judgement. That is, he is not making computations related to the wellbeing of people. Quite the contrary, he is completely ignoring the wellbeing of everyone but himself. Calling what he does a moral judgement obscures the issue. Similarly, when the pebblesorter says "It is good for pebbles to be sorted into prime numbered heaps" it is not making a moral judgement. It isn't doing computations about the wellbeing of people, it's doing computations about the numbers of pebbles. You, the sociopath, and the pebblesorter are not referring to the same concepts. You are referring to the wellbeing of people, the sociopath is referring to the gratification of his impulse, the pebblesorter is referring to the primality of pebble heaps. The phrase "moral judgement" should probably not be used to refer to all these different types of judgements, as they are not judgements about the same concepts at all. I would submit that if you removed the word "moral" and asked a pebblesorter "What action would best enhance the wellbeing of eudaemonic creatures" you and the pebblesorter would agree about quite a lot. The pebblesorter would then go back to sorting pebbles because it doesn't care about the wellbeing of eudaemonic creatures. (obviously this thought experiment would not work for a sociopath because sociopa
1TheOtherDave
For convenience, I am using the abbreviation "woec" for "wellbeing of eudaemonic creatures". I agree that if I asked a pebblesorter "What action would best enhance woec", assuming we could work out a shared definition of "eudaemonic", we would agree about quite a lot. If a pebblesorter asked me "What action would maximize prime-numbered heaps?" we would also agree about quite a lot. If we were to both answer the question "What action would optimize for my values?" our answers would be almost completely unrelated. I am willing to stop using the phrase "moral judgments" in this discussion to refer to judgments about what best implements the judger's values. This is entirely because disagreements about lexical usage are rarely productive when what we're really interested in is the referents. That said, I also prefer in that case to avoid using the phrase "moral judgments" to refer to judgments about what best achieves woec, since I don't actually use the phrase to mean that, which will get confusing. In fact, it's perhaps best to avoid the phrase altogether. I agree that a lot of disagreements about what action would best enhance woec, among creatures who value woec, is a blind men and the elephant problem. I agree that humans often want to do things that would not best enhance woec, even when we are aware that the thing we want to do would not best enhance woec. I agree that even among creatures who care about woec, there may not be agreement about values. I agree that when creatures whose values matter to me don't share values, I do well to embrace value pluralism.
-1Ghatanathoah
I am happy we are on the same page. If you really think that the phrase "moral judgements" is a useless and ambiguous phrase and that we shouldn't use it, I can respect that. But if enhancing weoc isn't what we should use the phrase "morality" to describe then what is? You also seem to nominate "optimizing for my values" as an alternative referent, but that doesn't seem right to me. Sociopaths are generally regarded as gravely immoral, even if they efficiently implement their values, because they don't care about the wellbeing of others. Should we really just jettison the word "morality" altogether? I suppose that could work. Since I've read Eliezer's work I've found that I can make the same points by substituting naturalistic statements for ones that use the word "moral." For instance, saying "The world would be a happier place if X didn't exist" is technically a naturalistic statement containing no value judgements, I use that a lot. But it seems like a shame to stop using such a powerful and effective word.
0TheOtherDave
It's not that I think the phrase is useless; it has many uses. It's that I think we use it to mean such different things I think this conversation is not well-served by introducing it. (You use it to refer to judgments related to weoc, I use it to refer to judgments related to the judger's values.) Yes, I would say that sociopaths make moral judgments, although their moral judgments differ from mine. I realize you would not say this, not because we disagree about sociopaths, but because we disagree about whether what sociopaths make can properly be labelled "moral judgment". I don't think the labeling question is terribly important or interesting. As you say, "moral" can usefully be cashed out in other terms.
0TheOtherDave
So, let us assume there exists some structure S3 in my head that implements my terminal values. Maybe that's eudaimonia, maybe that's Hungarian goulash, I don't really know what it is, and am not convinced that it's anything internally coherent (that is, I'm perfectly prepared to believe that my S3 includes mutually exclusive states of the world). I agree that when I label S3 "morality" I'm doing just what I do when I label S2 "eudaimonia" or label some other structure "prime". There's nothing special about the label "morality" in this sense. And if it turns out that you and I have close-enough S3s and we both label our S3s "morality," then we mean the same thing by "morality." Awesome. If, OTOH, I have S3 implementing my terminal values and you have some different structure, S4, which you also label "morality", then we might mean different things by "morality". Some day I might come to understand S3 and S4 well enough that I have a clear sense of the difference between them. At that point I have a lexical choice. I can keep associating S3 with the label "morality" or "right" and apply some other label to S4 (e.g., "pseudo-morality" or "nsheppard's right" or whatever). You might do the same thing. In that case, if (as you say) the only thing that might be remotely special about the label "morality" or "right" is that it might happens to refer to human terminal value, then it follows that there's nothing special about that label in that case, since in that case it doesn't refer to a common terminal value. It's just another word. Conversely, I can choose to associate the label "morality" or "right" with some new S5, a synthesis of S3 and S4... perhaps their intersection, perhaps something else. You might do the same thing. At that point we agree that "morality" means S5, even though S5 does not implement either of our terminal values.
2Ghatanathoah
The reason people think that Eliezer is really a relativist is that they see concepts like "good" and "right" as reducing down to mean, "the thing that I [the speaker, whoever it is] values." Eliezer is arguing that that is not what they reduce down to. He argues that "good" and "right" reduce down to something like "concepts related to enhancing the wellbeing of conscious eudaemonic life forms." It's not a trick of the language, Eliezer is arguing that "right" refers to [wellbeing related concept] and p-right refers to [primality sorting related concept]. The words "good" and "right" might be relative but the referent [wellbeing of conscious eudaemonic life forms] is not. The reason Eliezer focuses on fairness is that the concept of fairness is less nebulous than the concept of "right" so it is easier to see that it is not arbitrary. Pebble sorters and humans can both objectively agree on what it means to enhance the wellbeing of conscious eudaemonic life forms. Where they differ is whether they care about doing it. Pebble sorters don't care about the wellbeing of others. Why would they, unless it happened to help them sort pebbles? Similarly, humans and pebble sorters can both agree on which pebble heaps are prime-numbered. Where they differ is if they care about sorting pebbles. Humans don't care about pebble-sorting. Why would they, unless it helped then enhance the wellbeing of themselves and others? So if you define morality as "the thing that I care about," then I suppose it is relative, although I think that is not a proper use of the word "morality." But if you define it as "enhancing the wellbeing of eudaemonic life forms" then it is quite objective. Now, there might be room for moral disagreement in that people care about different aspects of wellbeing more. But that would be grounds for moral pluralism, not moral relativism. Regardless of what specific aspects of morality people focus on, certain things, like torturing the human population for all et
[-]Vladimir_Nesov40

Thanks, Yvain. Comparing well-understood special relativity to things characterized as "subjective" helps to clarify the sense in which they are really "objective", but look differently for different minds and are meaningless without any mind at all. You need a reference frame, and phenomenon does look different in different reference frames, but there are strict and consistent rules for converting between reference frames.

Reply
[-]Nick_Tarleton10

Very well said, Yvain!

Reply
[-]Nominull210

I think it's is more akin to saying that "easy" could just as well mean difficult in some alien language, and so words don't mean anything and language is a farce. That's the true linguistic relativist position.

Reply
[-]steven00

Yvain, I don't see why I would care about this thing you would call "moral", or refer to it often enough to justify such a short name.

Reply
[-]steven00

Or indeed why it's the same thing that people have traditionally meant by the word.

Reply
[-]J_Thomas200

People keep using the term "moral relativism". I did a Google search of the site and got a variety of topics with the term dating from 2007 and 2008. Here's what it means to me.

Relative moral relativism means you affirm that to the best of your knowledge nobody has demonstrated any sort of absolute morality. That people differ in moralities, and if there's anything objective to say one is right and another is wrong that you haven't seen it. That very likely these different moralities are good for different purposes and different circumstances, an... (read more)

Reply
[-]Z._M._Davis00

Submitted humbly for consideration: Ayn Rand is to libertarianism as Greg Egan is to transhumanism as Eliezer Yudkowsky is to moral relativism?

Reply
[-]Caledonian2-30

I don't think Eliezer belongs in the same category as Ayn Rand and Greg Egan.

Reply
[-]Russell_Wallace00

"But most of all - why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good?"

You are asking why anyone would choose life rather than what is good. Inclusive genetic fitness is just the long term form of life, as personal survival is the short-term form.

The answer is, of course, that one should not. By definition, one should always choose what is good. However, while there are times when it is right to give up one's life for a greater good, they are the exception. Most of the time, life is a subgoal of what is good, so there is no conflict.

Reply
[-]Erik300

A sidenote:

Eliezer: "It has something to do with natural selection never choosing a single act of mercy, of grace, even when it would cost its purpose nothing: not auto-anesthetizing a wounded and dying gazelle, when its pain no longer serves even the adaptive purpose that first created pain."

It always costs something; it is cheaper to build a gazelle that always feels pain than one that does so until some conditions are met. This is the related to the case of supposing a spaceship that has passed out of your lightcone still exists.

Natural select... (read more)

Reply
[-]Roko10

Yvian: "So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is h-right."

- well said. Modulo Eliezer's lack of explicitness about his definition of "h-right", I fail to see how the human perspective could be anything other than h-right. This post is just an applause light for the values that we currently like, and I think that that is a bad sign.

If human values were so great, you wouldn't have to artificially make them look better by saying things like

"So which... (read more)

Reply
[-]Roko00

Z.M Davis: "Submitted humbly for consideration: Ayn Rand is to libertarianism as Greg Egan is to transhumanism as Eliezer Yudkowsky is to moral relativism?"

- not sure I get this... Rand abhorred libertarianism because she thought it was half-baked and amateurish, but actually she is a libertarian, Egan spoke out against transhumanism because, uuum, he thinks we're all crackpots, but actually he's a transhumanist, Yudkowsky speaks out against moral relativism but actually he's the canonical example of a relativist. Ah, yes, ok.

Spot on, Z.M, Seconded.

Reply
[-]Kip_Werking20

Michael Anissimov, August 14, 2008 at 10:14 PM asked me to expound.

Sure. I don't want to write smug little quips without explaining myself. Perhaps I'm wrong.

It's difficult to engage Eliezer in debate/argument, even in a constructive as opposed to adversarial way, because he writes so much material, and uses so many unfamiliar terms. So, my disagreement may just be based on an inadequate appreciation of his full writings (e.g. I don't read every word he posts on overcomingbias; although I think doing so would probably be good for my mind, and I eagerly ... (read more)

Reply
[-]Virge200

Kip Werking: "P2. But, all we have to prove that giving to charity, etc., is right, is that everyone thinks it is"

You're stating that there exists no other way to prove that giving to charity is right. That's an omniscient claim.

Still, it's unlikely to be defeated in the space of a comment thread, simply because your sweeping generalization about the goodness of charity is far from being universally accepted. A very general claim like that, with no concrete scenario, no background information on where it is to be applied, makes relativism a fore... (read more)

Reply
[-]Alberto_Gómez00

I think that all the morality is is just to be sure that the persons close to me rank high in the prisioner dilemma game, and to assure others that I rank high too. Even higuer than I really am.

For this purpose is all that has been done by evolution, intellectual and religious thinking.

Reply
[-]J_Thomas200

The same boy who rationalized a way into believing there was a chocolate cake in the asteroid belt, should know better than to rationalize himself into believing it is right to prefer joy over sorrow.

Obviously, he does know. So the next question is, why does he present material that he knows is wrong?

Professional mathematicians and scientists try not to do that because it makes them look bad. If you present a proof that's wrong then other mathematicians might embarrass you at parties. But maybe Eliezer is immune to that kind of embarrassment. Socrates pres... (read more)

Reply
[-]minnmass60

I do apologize for coming late to the party; I've been reading, and really feel like I'm missing an inferential step that someone can point me towards.

I'll try to briefly summarize, knowing that I'll gloss over some details; hopefully, the details so glossed over will help anyone who wishes to help me find the missing step.

It seems to me that Eliezer's philosophy of morality (as presented in the metaethics sequence) is: morality is the computation which decides which action is right (or which of N actions is the most right) by determining which action maxi... (read more)

Reply
[-]Rain120

My first problem (which may well be a missed inferential step) is with the assumed universality, within humanity, of a system of goals.

From what I've seen, others have the same objection; I do as well, and I have not seen an adequate response.

how is it that humans have discovered "right" while the Pebble-people have discovered only "p-right"? Even if I grant the assertion that all humans are using the same fundamental morality, and Alice and Bob would necessarily agree if they had access to the same information, how is it that humans have discovered "right" and not "h-right"?

From what I understand, everyone except Eliezer is more likely to hold the view that he found "h-right", but he seems unwilling to call it that even when pressed on the matter. It's another point on which I agree with your confusion.

as I understand it, Eliezer's morality simply says "do whatever the computation tells you to do" without offering any help on what that computation actually looks like

We don't have quite the skill to articulate it just yet, but possibly AI and neuroscience will help. If not, we might be in trouble.

As I said, I

... (read more)
Reply
6Kutta
"Better" and "closer to morality" and "h-morality" refer to the same thing here. "H-morality is better" roughly means "better is better". Seeing no evidence that h-morality is better is like seeing no evidence that 2=2. As far as I can see this is a reason why Eliezer doesn't bother with calling morality "h-morality" though I might be erring.
[-]TheOtherDave40

OK... let me see if I'm following.

The idea, trying to rely on as few problematic words as possible, is:

  • There exists a class of computations M which sort proposed actions/states/strategies into an order, and which among humans underly the inclination to label certain actions/states "good", "bad", "right", "wrong", "moral", etc.(4) For convenience I'll label the class of all labels like that "M-labels."

  • If two beings B1 and B2 don't implement (1) a common M-instance, M-labels may not be meanin

... (read more)
Reply
[-]buybuydandavis00

So I really must deny the charges of moral relativism: I don't think that human morality is arbitrary at all, and I would expect any logically omniscient reasoner to agree with me on that. We are better than the Pebblesorters, because we care about sentient lives, and the Pebblesorters don't. Just as the Pebblesorters are p-better than us, because they care about pebble heaps, and we don't. Human morality is p-arbitrary, but who cares? P-arbitrariness is arbitrary.

Is the Logically Omniscient Reasoner agreeing that human morality is not h-arbitrary, or that it is not lor-arbitrary?

How do we know that The LOR (ha! walked into that one) isn't a Pebblesorter?

Reply
[-]Lukas_Gloor10

What p-bothers me (sorry couldn't resist!) about this approach is that "rightness" nowhere explicitly refers to "others", i.e. other conscious beings / consciousness-moments. Isn't there an interesting difference between a heap of eight pebbles (very p-bad) and a human getting tortured (very bad)? Concerning the latter, we can point to that human's first-person-perspective directly evaluating its current conscious state and concluding that the state is bad, i.e. that the person wants to get the hell out of it. This is a source of disval... (read more)

Reply
0PrawnOfFate
If you define "morality" broadly, as maximising values you can end up with that sort of thing. Some would take the attitude that if your definitions covers counterintuitive cases, your definition is too broad.
[-]Will_Lugar00

You, the human, might say we really should pursue beauty and laughter and love (which is clearly very important), and that we p-should sort pebbles (but that doesn't really matter). And that our way of life is really better than the Pebblesorters, although their way of life has the utterly irrelevant property of being p-better.

But the Pebblesorters would say we h-should pursue beauty and laughter and love (boring!), and that we really should sort pebbles (which is the self-evident meaning of life). Further, they will say their way of life is really better ... (read more)

Reply
Moderation Log
119Comments