Continuation ofIs Morality Preference?

(Disclaimer:  Neither Subhan nor Obert represent my own position on morality; rather they represent different sides of the questions I hope to answer.)

Subhan:  "What is this 'morality' stuff, if it is not a preference within you?"

Obert:  "I know that my mere wants, don't change what is right; but I don't claim to have absolute knowledge of what is right—"

Subhan:  "You're not escaping that easily!  How does a universe in which murder is wrong, differ from a universe in which murder is right?  How can you detect the difference experimentally?  If the answer to that is 'No', then how does any human being come to know that murder is wrong?"

Obert:  "Am I allowed to say 'I don't know'?"

Subhan:  "No.  You believe now that murder is wrong.  You must believe you already have evidence and you should be able to present it now."

Obert:  "That's too strict!  It's like saying to a hunter-gatherer, 'Why is the sky blue?' and expecting an immediate answer."

Subhan:  "No, it's like saying to a hunter-gatherer:  Why do you believe the sky is blue?"

Obert:  "Because it seems blue, just as murder seems wrong.  Just don't ask me what the sky is, or how I can see it."

Subhan:  "But—aren't we discussing the nature of morality?"

Obert:  "That, I confess, is not one of my strong points.  I specialize in plain old morality.  And as a matter of morality, I know that I can't make murder right just by wanting to kill someone."

Subhan:  "But if you wanted to kill someone, you would say, 'I know murdering this guy is right, and I couldn't make it wrong just by not wanting to do it.'"

Obert:  "Then, if I said that, I would be wrong.  That's common moral sense, right?"

Subhan:  "Argh!  It's difficult to even argue with you, since you won't tell me exactly what you think morality is made of, or where you're getting all these amazing moral truths—"

Obert:  "Well, I do regret having to frustrate you.  But it's more important that I act morally, than that I come up with amazing new theories of the nature of morality.  I don't claim that my strong point is in explaining the fundamental nature of morality.  Rather, my strong point is coming up with theories of morality that give normal moral answers to questions like, 'If you feel like killing someone, does that make it right to do so?'  The common-sense answer is 'No' and I really see no reason to adopt a theory that makes the answer 'Yes'.  Adding up to moral normality—that is my theory's strong point."

Subhan:  "Okay... look.  You say that, if you believed it was right to murder someone, you would be wrong."

Obert:  "Yes, of course!  And just to cut off any quibbles, we'll specify that we're not talking about going back in time and shooting Stalin, but rather, stalking some innocent bystander through a dark alley and slitting their throat for no other reason but my own enjoyment.  That's wrong."

Subhan:  "And anyone who says murder is right, is mistaken."

Obert:  "Yes."

Subhan:  "Suppose there's an alien species somewhere in the vastness of the multiverse, who evolved from carnivores.  In fact, through most of their evolutionary history, they were cannibals.  They've evolved different emotions from us, and they have no concept that murder is wrong—"

Obert:  "Why doesn't their society fall apart in an orgy of mutual killing?"

Subhan:  "That doesn't matter for our purposes of theoretical metaethical investigation.  But since you ask, we'll suppose that the Space Cannibals have a strong sense of honor—they won't kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong.  Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies.  But so far as the actual killing is concerned, the aliens just think it's fun.  When someone gets executed for, say, driving through a traffic light, there's a bidding war for the rights to personally tear out the offender's throat."

Obert:  "Okay... where is this going?"

Subhan:  "I'm proposing that the Space Cannibals not only have no sense that murder is wrong—indeed, they have a positive sense that killing is an important part of life—but moreover, there's no path of arguments you could use to persuade a Space Cannibal of your view that murder is wrong.  There's no fact the aliens can learn, and no chain of reasoning they can discover, which will ever cause them to conclude that murder is a moral wrong.  Nor is there any way to persuade them that they should modify themselves to perceive things differently."

Obert:  "I'm not sure I believe that's possible—"

Subhan:  "Then you believe in universally compelling arguments processed by a ghost in the machine.  For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value—1.  A mind is a physical device and you can't have a little blue woman pop out of nowhere and make it say 1 when the physics calls for it to say 0."

Obert:  "Suppose I were to concede this.  Then?"

Subhan:  "Then it's possible to have an alien species that believes murder is not wrong, and moreover, will continue to believe this given knowledge of every possible fact and every possible argument.  Can you say these aliens are mistaken?"

Obert:  "Maybe it's the right thing to do in their very different, alien world—"

Subhan:  "And then they land on Earth and start slitting human throats, laughing all the while, because they don't believe it's wrong.  Are they mistaken?"

Obert:  "Yes."

Subhan:  "Where exactly is the mistake?  In which step of reasoning?"

Obert:  "I don't know exactly.  My guess is that they've got a bad axiom."

Subhan:  "Dammit!  Okay, look.  Is it possible that—by analogy with the Space Cannibals—there are true moral facts of which the human species is not only presently unaware, but incapable of perceiving in principle?  Could we have been born defective—incapable even of being compelled by the arguments that would lead us to the light?  Moreover, born without any desire to modify ourselves to be capable of understanding such arguments?  Could we be irrevocably mistaken about morality—just like you say the Space Cannibals are?"

Obert:  "I... guess so..."

Subhan:  "You guess so?  Surely this is an inevitable consequence of believing that morality is a given, independent of anyone's preferences!  Now, is it possible that we, not the Space Cannibals, are the ones who are irrevocably mistaken in believing that murder is wrong?"

Obert:  "That doesn't seem likely."

Subhan:  "I'm not asking you if it's likely, I'm asking you if it's logically possible!  If it's not possible, then you have just confessed that human morality is ultimately determined by our human constitutions.  And if it is possible, then what distinguishes this scenario of 'humanity is irrevocably mistaken about morality', from finding a stone tablet on which is written the phrase 'Thou Shalt Murder' without any known justification attached?  How is a given morality any different from an unjustified stone tablet?"

Obert:  "Slow down.  Why does this argument show that morality is determined by our own constitutions?"

Subhan:  "Once upon a time, theologians tried to say that God was the foundation of morality.  And even since the time of the ancient Greeks, philosophers were sophisticated enough to go on and ask the next question—'Why follow God's commands?'  Does God have knowledge of morality, so that we should follow Its orders as good advice?  But then what is this morality, outside God, of which God has knowledge?  Do God's commands determine morality?  But then why, morally, should one follow God's orders?"

Obert:  "Yes, this demolishes attempts to answer questions about the nature of morality just by saying 'God!', unless you answer the obvious further questions.  But so what?"

Subhan:  "And furthermore, let us castigate those who made the argument originally, for the sin of trying to cast off responsibility—trying to wave a scripture and say, 'I'm just following God's orders!'  Even if God had told them to do a thing, it would still have been their own decision to follow God's orders."

Obert:  "I agree—as a matter of morality, there is no evading of moral responsibility.  Even if your parents, or your government, or some kind of hypothetical superintelligence, tells you to do something, you are responsible for your decision in doing it."

Subhan:  "But you see, this also demolishes the idea of any morality that is outside, beyond, or above human preference.  Just substitute 'morality' for 'God' in the argument!"

Obert:  "What?"

Subhan:  "John McCarthy said:  'You say you couldn't live if you thought the world had no purpose. You're saying that you can't form purposes of your own-that you need someone to tell you what to do. The average child has more gumption than that.'  For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question:  'And why obey this morality?'  It would be your decision to follow this trend of the universe, or obey this structure of logic.  Your decision—and your preference."

Obert:  "That doesn't follow!  Just because it is my decision to be moral—and even because there are drives in me that lead me to make that decision—it doesn't follow that the morality I follow consists merely of my preferences.  If someone gives me a pill that makes me prefer to not be moral, to commit murder, then this just alters my preference—but not the morality; murder is still wrong.  That's common moral sense—"

Subhan:  "I beat my head against my keyboard!  What about scientific common sense?  If morality is this mysterious given thing, from beyond space and time—and I don't even see why we should follow it, in that case—but in any case, if morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

Obert:  "Coincidence?  How so?"

Subhan:  "Just where on Earth do you think the emotion of love comes from?  If the ancient Greeks had ever thought of the theory of natural selection, they could have looked at the human institution of sexual romance, or parental love for that matter, and deduced in one flash that human beings had evolved—or at least derived tremendous Bayesian evidence for human evolution.  Parental bonds and sexual romance clearly display the signature of evolutionary psychology—they're archetypal cases, in fact, so obvious we usually don't even see it."

Obert:  "But love isn't just about reproduction—"

Subhan:  "Of course not; individual organisms are adaptation-executers, not fitness-maximizers.  But for something independent of humans, morality looks remarkably like godshatter of natural selection.  Indeed, it is far too much coincidence for me to credit.  Is happiness morally preferable to pain?  What a coincidence!  And if you claim that there is any emotion, any instinctive preference, any complex brain circuitry in humanity which was created by some external morality thingy and not natural selection, then you are infringing upon science and you will surely be torn to shreds—science has never needed to postulate anything but evolution to explain any feature of human psychology—"

Obert:  "I'm not saying that humans got here by anything except evolution."

Subhan:  "Then why does morality look so amazingly like a product of an evolved psychology?"

Obert:  "I don't claim perfect access to moral truth; maybe, being human, I've made certain mistakes about morality—"

Subhan:  "Say that—forsake love and life and happiness, and follow some useless damn trend of the universe or whatever—and you will lose every scrap of the moral normality that you once touted as your strong point.  And I will be right here, asking, 'Why even bother?'  It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow."

Obert:  "All right... then maybe the reason morality seems to bear certain similarities to our human constitutions, is that we could only perceive morality at all, if we happened, by luck, to evolve in consonance with it."

Subhan:  "Horsemanure."

Obert:  "Fine... you're right, that wasn't very plausible.  Look, I admit you've driven me into quite a corner here.  But even if there were nothing more to morality than preference, I would still prefer to act as morality were real.  I mean, if it's all just preference, that way is as good as anything else—"

Subhan:  "Now you're just trying to avoid facing reality!  Like someone who says, 'If there is no Heaven or Hell, then I may as well still act as if God's going to punish me for sinning.'"

Obert:  "That may be a good metaphor, in fact.  Consider two theists, in the process of becoming atheists.  One says, 'There is no Heaven or Hell, so I may as well cheat and steal, if I can get away without being caught, since there's no God to watch me.'  And the other says, 'Even though there's no God, I intend to pretend that God is watching me, so that I can go on being a moral person.'  Now they are both mistaken, but the first is straying much further from the path."

Subhan:  "And what is the second one's flaw?  Failure to accept personal responsibility!"

Obert:  "Well, and I admit I find that a more compelling argument than anything else you have said.  Probably because it is a moral argument, and it has always been morality, not metaethics, with which I claimed to be concerned.  But even so, after our whole conversation, I still maintain that wanting to murder someone does not make murder right.  Everything that you have said about preference is interesting, but it is ultimately about preference—about minds and what they are designed to desire—and not about this other thing that humans sometimes talk about, 'morality'.  I can just ask Moore's Open Question:  Why should I care about human preferences?  What makes following human preferences rightBy changing a mind, you can change what it prefers; you can even change what it believes to be right; but you cannot change what is right.  Anything you talk about, that can be changed in this way, is not 'right-ness'."

Subhan:  "So you take refuge in arguing from definitions?"

Obert:  "You know, when I reflect on this whole argument, it seems to me that your position has the definite advantage when it comes to arguments about ontology and reality and all that stuff—"

Subhan:  "'All that stuff'?  What else is there, besides reality?"

Obert:  "Okay, the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks.  But I still think the morality-as-given viewpoint has the advantage when it comes to, you know, the actual morality part of it—giving answers that are good in the sense of being morally good, not in the sense of being a good reductionist.  Because, you know, there are such things as moral errors, there is moral progress, and you really shouldn't go around thinking that murder would be right if you wanted it to be right."

Subhan:  "That sounds to me like the logical fallacy of appealing to consequences."

Obert:  "Oh?  Well, it sounds to me like an incomplete reduction—one that doesn't quite add up to normality."

 

Part of The Metaethics Sequence

Next post: "Where Recursive Justification Hits Bottom"

Previous post: "Is Morality Preference?"

New to LessWrong?

New Comment
100 comments, sorted by Click to highlight new comments since: Today at 5:22 AM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Subhan: "You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No', then how does any human being come to know that murder is wrong?" ... Obert: "Because it seems blue, just as murder seems wrong. Just don't ask me what the sky is, or how I can see it."

But we already know why murder seems wrong to us. It's completely explained by a combination of game theory, evolutionary psychology, and memetics. These explanations screen off our apparent moral perceptions from any other influence. In order words, conditioned on these explanations being true, our moral perceptions are independent of (i.e. uncorrelated with) any possible morality-as-given, even if it were to exist.

So there is a stronger argument against Obert than the one Subhan makes. It's not just that we don't know how we can know about what is right, but rather that we know we can't know, at least not through these apparent moral perceptions/intuitions.

-1MugaSofer11y
He does, in fact, point out that if morality came from evolution (which our minds came from evolution, then our moral preferences must have too, so why would evolution coincidentally choose values that correspond to some Universal Imperative?

As with human aesthetic sense, human morality may be approximations and versions of more absolutely definable optimal solutions to information-theoretic, game-theoretic, social, economic, intelligence, signaling, cooperation problems. Therefore it may be likely that an alien race could share some of the same values as we do, because it may turn out that they are "good" solutions for intelligent culture bearing species in general. But there is nothing in the universe in it self that says that theese optimal solutions, or any value what-so-ever can... (read more)

And to answer Obert's objection that Subhan's position doesn't quite add up to normality: before we knew game theory, evolutionary psychology, and memetics, nothing screened off our moral perceptions/intuitions from a hypothesized objective moral reality, so that was perhaps the best explanation available, given what we knew back then. And since that was most of human history, it's no surprise that morality-as-given feels like normality. But given what we know today, does it still make sense to insist that our meta-theory of morality add up to that normality?

I will try to express some of my points more accurately... A human value, may it concern, knowledge, morality or beauty gets it's meaning from it's emotional base although it may be a frequent value in the space of possible intelligent species. Only minds can attribute value to something. The thing it attributes it to, may be universal or specific, but the thing itself is can not be valued by something other than a mind. To value something is a cognitive, emotional process, not some intrinsic property of some phenomenon. But to believe this as the mind you... (read more)

I think that we need a much better explanation of this word "mind". Supposedly mind space contains a -1 for every 1, but that simply sounds like system space. I honestly think that the ontology has to go deeper here before progress is possible. Similar problem to born postulates and why we aren't Boltzman Brains.

Similar problem to born postulates and why we aren't Boltzman Brains.
We are Boltzmann Brains. You simply don't appreciate what restrictions are inherent in specifying the subset of Brains that can be called "we".

Not that this has anything to do with the topic, which everyone is very carefully skating around without addressing: what are operational definitions for right and wrong? When Obert says "Because it seems blue, just as murder seems wrong.", what collection of properties does wrong refer to? For that matter, what does blue ... (read more)

I don't think you have to postulate Space Cannibals in order to imagine rational creatures who don't think murder is wrong. For a recent example, consider Rwanda 1994.

And I think it's quite possible that there might exist moral facts which humans are incapable of perceiving. We aren't just universal Turing machines, after all. Billions of years of evolution might produce creatures with moral blind spots, anologous to the blind spot in the human eye. Just as the squid's eye has no blind spot, a different evolutionary path might produce creatures with a greater or lesser innate capacity to perceive goodness than ourselves.

Maybe this will make it easier:

Obert says "just as murder seems wrong". There is a redundancy in that phrase. What is the redundancy, and why doesn't Obert perceive it as one?

What is the difference between saying something is a rube and not a blegg, and saying that someone appears to be a rube and not a blegg?

What is the difference between saying something is imperceivable, and saying something appears to be imperceivable?

(Subhan wrote:) "And if you claim that there is any emotion, any instinctive preference, any complex brain circuitry in humanity which was created by some external morality thingy and not natural selection, then you are infringing upon science and you will surely be torn to shreds - science has never needed to postulate anything but evolution to explain any feature of human psychology -" Subhan: "Suppose there's an alien species somewhere in the vastness of the multiverse, who evolved from carnivores. In fact, through most of their evoluti... (read more)

0[anonymous]8y
There are a lot of clever ideas in this post, despite the harsh downvotes. You may have some misgivings about the extent to which say, mental health issues may be a barrier to security clearances. It's more like people disqualify themselves by lying or failing to apply in the first place. Those who do get through and get issues, are prisoners of their own misconceptions. Austalia's protective security guidelines are based around subjective evaluations of see this. Caution, if you're speaked by getting tracked, note that this is a word document on a Aus gov website. It also explicitly says that seeking help from mental health places shouldn't be the sole basis of exclusion, and the guidelines suggest that the opinion of a mental health professional should be given due consideration. This wasn't always the way things were down, at least in the us. The really contentious issue here is whether it is correct to privellage the hypothesis that those seeking mental health care are more likely to have worse judgment, reliability, or trustworthiness. Intuitions and stereotypes say yes. Research suggests they among those seeking treatment, they are not anymore violent, I'm not sure about those criteria specifically, but I suspect that there is far too much assumption of mental illness as a description of abberant behaviour, rather than as an exclusive construct resilient to black swans and that soon mental health and the military and intelligence fields will become subject to scrutinty by mental health activists, the same way other activists have scrutinised discrimination in security fields.

The notion of morality as subjectively objective computation seems a lot closer to Subhan's position than Obert's.

Yes, EY's past positions about Morality are closer to Subhan's than Obert's. But AGI is software programming and hardware engineering, not being a judge or whoever writes laws. I wouldn't suggest deifying EY if your goal is to learn ethics.

But we already know why murder seems wrong to us. It's completely explained by a combination of game theory, evolutionary psychology, and memetics. These explanations screen off our apparent moral perceptions from any other influence. In order words, conditioned on these explanations being true, our moral perceptions are independent of (i.e. uncorrelated with) any possible morality-as-given, even if it were to exist.

Let's try the argument with mathematics: we know why we think 5 is a prime number. It's completely explained by our evolution, experiences, an... (read more)

"But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure.

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

"You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No'...

Minor quibble - 'no' is not a sensical answer to any of those questions. Possibly remove the word 'how' from one of them?

Once again, no revelations that I haven't come across on my own, but crystallised and clarified brilliantly. Looking forward to the next few.

It seems to me that Obert makes a faulty interpretation of "there is no reason to talk about a 'morality' distinct from what people want.", but i would like to know what the author thinks. In my view, that assertion says not that ALL MORAL CLAIMS ARE WHIMS, but instead that to understand and parse and compare moral claims we have to resort to wants. In other words, that WANTS ARE THE OBJECT OF MORALITY, THOUGH NOT IT'S MATTER. To understand any moral claim we have to consider how it imparts onto what real, concrete persons feel and desire.

"I want pie" and "I deserve pie" are different, but i don't see how Subhan's arguments aspire to make them equal.

Obert's arguments seem much closer to "how it feels from the inside", Subhan in general does seem to have stronger actual arguments, however:

"For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question: 'And why obey this morality?'" This, to me, smells of zombieism. "for any configuration of matter/energy/whatever, we can ask 'and why should we believe that this is actually conscious rather than just a structure immitating a consciousness?'"

(ZMDavis wrote:) "But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure."

Argh. I didn't mean that as a critique on EY's prowess as an AGI theorist or programmer. I doubt Jesus would've wanted people to deify him, just to be nice to eachother. I doubt EY meant for his learning of philosophy to be interpreted as some sort of Moral code, he was just arrogant enough not to state he was sometimes using his list to as a tool to develo... (read more)

So here's a question Eliezer: is Subhan's argument for moral skepticism just a concealed argument for universal skepticism? After all, there are possible minds that do math differently, that do logic differently, that evaluate evidence differently, that observe sense-data differently...

Either Subhan can distinguish his argument from an argument for universal skepticism, or I say that it's refuted by reductio, since universal skepticism fails to the complete impossibility of asserting it consistently + things like moorean facts.

Phillip, you're the one who brought up "deification," in response to my one-line comment, which you seem to have read a lot into. My second comment was intended to be humorous. I apologize for the extent to which I contributed to this misunderstanding.

Eliezer seems to suggest that the only possible choices are morality-as-preference or morality-as-given, e.g. with reasoning like this:

[...] the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks. But I still think the morality-as-given viewpoint has the advantage [...]

But really, evolutionary psychology, plus some kind of social contract for group mutual gain, seems to account for the vast bulk of what people consider to be "moral" actions, as well as the conflict between private individual desires vs. a... (read more)

I think it's probably useful to taboo the word "should" for this discussion. I think when people say you "should" do X rather than Y it means something like "experience indicates X is more likely to lead to a good outcome than Y". People tend to have rule-based rather than consequence based moral systems because the full consequences of one's actions are unforeseeable. A rule like "one shouldn't lie" comes about because experience has shown that lying often has negative consequences for the speaker and listener and p... (read more)

Relationships are real. For example if a plant is "under" a table, that is a fact, not a subjective whim of the observer. So if morality is a relationship, then aliens and man can have different moralities but both be objective, not subjective. The relationship would be between the object sought and the entity seeking it, e.g. murder + man = bad, murder + alien = good.

-3Peterdjones11y
Correct. Morality is a function from facts about ones community's preferences to norms of behaviour. Plug in different facts, and you get different norms een if the function is essentially the same.

Paul Gowder,

Yes, there are possible minds that do math/logic/deduction differently. Most of these logically possible minds perform even worse than humans in these aspects, and would die out.

In this universe, if one wishes to reach ones goals, one has to choose to (try to) do math/logic/deduction in the correct way; the way that delivers results. What works is determined by the laws of physics and logic that in our universe seem quite coherent and understandable (to a degree, at least).

There's no reason to be skeptical about whether I actually have some goa... (read more)

Subhan's question here, "How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally?" is such a gem.

I wonder if Eliezer intended it as parody.

If somebody said to me "morality is just what we do." If they presented evidence that the whole apparatus of their moral philosophy was a coherent description of some subset of human psychology and sociology. Then that would be enough for me. It's just a description of a physical system. Human morality would be what human animals do. Moral responsibility wouldn't be problematic; moral responsibility could be as physical as gravity if it were psychologically and sociologically real. "I have a moral responsibility" would be akin to "... (read more)

I've thought about Space Cannibals and the like before (i.e. creatures that kill one of the sexes during sexual reproduction). My suspicion is that even if such creatures evolved and survived, by the time they had a civilization, many would be saying to one another, "There really should be a better way..."

Evidence for this is the fact that even now, there are many human beings claiming it is wrong to kill other animals, despite the fact that humans evolved to kill and eat other animals. Likewise, in the ancestral environment, various tribes usual... (read more)

Robin, As Eliezer has pointed out, evolution is a nonhuman optimizer which is in many ways more powerful than the human mind. On the assumption that humans have a moral sense, I don't think we should expect to be able to understand why. That might simply be a problem which is too difficult for people to solve. That aside, a man's virtues benefit the society he lives in; his inclination to punish sin will encourage others to act virtuously as well. If his society is a small tribe of his relatives, then even the weaker forms of kin selection theory can explain the benefit of knowledge of good and evil.

Treating those who do not deserve respect with respect is basically spitting on those who do deserve it, especially those who work hard for it. I think you need to treat those you don't know with the "presumption of respect"; that is, if you don't know that they don't deserve it, assume they do. Borrowed from Smith's "presumption of rationality"; when you argue with someone, assume that they are rational until they demonstrate otherwise.

Richard, would you accept the same argument about God, that we know there is a God but don't really understand how we know, but gosh darn it we feel like there must be one so there must be one? Yes we evolved to help kin, and we expect many but hardly all other species to do this as well. But unless we know whether that behavior is moral we don't know if that is a process that makes our moral intuitions correlate with moral truth.

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

Let's detail the explanation for light to see if we can find a parallel explanation for morality. Brief explanation for light: light bounces off things in the environment in a way which can in principle be used to draw correct inferences about distant objects in the environment. Eventually, some animals evolve a mechanism for doing just this.

Let's attempt the same for ... (read more)

Robin,

Our moral intuitions correspond with moral truths for much the same reason that our rational predictions correspond with more concrete physical truths. A man who ignores reason will stick his hand back in the fire after being burned the first time. Such behavior will kill him, probably sooner rather than later. An man who is blind to good and evil may do quite well for himself, but a society whose citizens ignore virtue will suffer approximately the same fate as the twice-burned fool.

Richard, I agree that some social norms help a society prosper while others can "burn" it. And we have the intuition that morally right acts correspond to social norms that help societies prosper. But we would have had that intuition even if morally right acts had corresponded to the opposite. What evolutionary pressure could have produced the correct intuitions about this meta question?

Constant, I would say that objective illness is just as problematic as objective morality; it's just less obviously problematic because in everyday contexts, we're more used to dealing with disputes about morality than about illness. You mention that "if we select an ill partner for producing offspring we may produce no offspring," and in an evolutionary context, probably we could give some fitness-based account of illness. However, this evolutionary concept of "illness" cannot be the ordinary meaning of the word, because no one actual... (read more)

Constant wrote: So one place where one could critique your argument is in the bit that goes: "conditioned on X being the case, then our beliefs are independent of Y". The critique is that X may in fact be a consequence of Y, in which case X is itself not independent of Y.

Good point, my argument did leave that possibility open. But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

So if I were to draw ... (read more)

I wonder if Eliezer intended it as parody.
He'd be making a serious mistake if so.

Eliezer: You have perhaps already considered this, but I think it would be helpful to learn some lessons from E-Prime when discussing this topic. E-Prime is a subset of English that bans most varieties of the verb "to be".

I find sentences like "murder is wrong" particularly underspecified and confusing. Just what, exactly, is meant by "is", and "wrong"? It seems like agreeing on a definition for "murder" is the easy part.

It seems the ultimate confusion here is that we are talking about instrumental values (... (read more)

My earlier comment is not to imply that I think "maximization of human happiness" is the most preferred goal.

An easily obvious one, yes. But faulty; "human" is a severely underspecified term.

In fact, I think that putting in place a One True Global Goal would require ultimate knowledge about the nature of being, to which we do not have access currently.

Possibly, the best we can do is come up with plausible global goal that suits us for medium run, while we try to find out more.

That is, after all, what we have always done as human beings.

Wanting to murder doesn't make it right. Nothing makes anything morally right.

Robin,

I don't understand your counterfactual.

"Good" and "Evil" are the names for what people perceive with their moral sense. I think we've agreed that this perception correlates to something universally observable (namely, social survival), so these labels are firmly anchored in the physical world. It looks to me like you're trying to assign these names to something else altogether (namely, something which does not correlate with human moral intuitions), and it's not clear to me how this makes sense.

Richard, if morality just meant social norms that help societies prosper, then of course we have little problem understanding how the two could be correlated, and how we could come to know about them. But if morality means something else, then we face the much harder question of how it is we could know about this something else.

For those impatient to know where Eliezer is going with this series, it looks like he gaves us a sneak preview a little more than a year ago. The answer is morality-as-computation.

Eliezer, hope I didn't upset your plans by giving out the ending too early. When you do get to morality-as-computation, can you please explain what exactly is being computed by morality? You already told us what the outputs look like: "Killing is wrong" and "Flowers are beautiful", but what are the inputs?

EY: "human cognitive psychology has not had time to change evolutionarily over that period"

Under selective pressures, human populations can and have significantly changed in less than two thousand years. Various behavioral traits are highly heritable. Genghis Khan spread his behavioral genotype throughout Asia. (For this discussion this is a nitpick but I dislike seeing false memes spread.)

re: FAI and morality

From my perspective morality is a collection of rules that make cooperative behavior beneficial. There are some rules that should apply to ... (read more)

Robin,

I don't know how people are capable of discerning moral truths. I also don't know how people are capable of discerning scientific or mathematical truths. It seems to me that these are similar capabilities, and the one is no more suprising or unlikely than the other.

Richard, while there are surely many details we would like to understand better, surely we understand the basic outline of how we discern scientific and mathematical truths. For example, in math we use contradiction to eliminate possible implications of axiom sets, and in science we use empirical results to eliminate possible abstract theories. We have nothing remotely similar in morals. You never said whether you approved of a similar argument about knowledge of God.

Z. M. Davis writes: ... objective illness is just as problematic as objective morality

I would argue that to answer Robin's challenge is not necessarily to assert that there is such a thing as objective illness.

Accounts have been given of the pressure producing the ability to see beauty (google sexual selection or see e.g. this). This does not require that there is some eternal beauty written in the fabric of the universe - it may be, for example, that each species has evolved its own standard of beauty, and that selection is operating on both sides, i.e., ... (read more)

"what possible pressure could produce an ability to see morality?"

Unlike the other Richard, I don't think we "see" morality with a special "sense", or anything like that. But if we instead understand morality as a rational idealization, building on our perfectly ordinary general capacity for systematizing judgments so as to increase their overall coherence (treating like cases alike, etc.), then there's no great mystery here.

Dynamically Linked writes: But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

According to Tegmark "there is only mathematics; that is all that exists". Suppose he is right. Then moral truths, if there are any, are (along with all other truths) mathematical truths. Unless you presuppose that moral truths cannot be mathematical truths then you have not ruled out moral truths when you say that so-and... (read more)

Robin:

Discarding false mathematical and scientific conjectures is indeed much easier than discarding false moral conjectures. However, as Eliezer pointed out in an earlier post, a scientist who can come up with a hypothesis that has a 10% chance of being true has already gone most of the way from ignorance to knowledge. I would argue that hypothesis generation is a poorly-understood nonrational process in all three cases. A mathematician who believes he has found truth can undertake the further steps of writing a formal proof and submitting his work to ... (read more)

Constant, if moral truths were mathematical truths, then ethics would be a branch of mathematics. There would be axiomatic formalizations of morality that do not fall apart when we try to explore their logical consequences. There would be mathematicians proving theorems about morality. We don't see any of this.

Isn't it simpler to suppose that morality was a hypothesis people used to explain their moral perceptions (such as "murder seems wrong") before we knew the real explanations, but now we find it hard to give up the word due to a kind of memetic inertia?

Constant, if moral truths were mathematical truths, then ethics would be a branch of mathematics. There would be axiomatic formalizations of morality that do not fall apart when we try to explore their logical consequences. There would be mathematicians proving theorems about morality. We don't see any of this.

If Tegmark is correct, then everything is mathematics. Do you dispute Tegmark's claim that "there is only mathematics; that is all that exists"? Do you think your argument is any good against Tegmark's hypothesis? Will you tell Tegmark, &qu... (read more)

What's so bad about morality being a mere human's construct - in other words, the notion that there is no "stone tablet" of morals? In fact, I think the notion that morality exists objectively, like some looming Platonic figure, raises more questions than would be solved by such a condition.

I think the best way to construct this "morality" is just to say that it's got a quasi-mathematical existence, it's axiomic, and it's all augmented by empirical/logical reasoning.

Why accept it, why be moral? I feel the same way about this question as I do about the question of why somebody who believes "if A, then B," and also believes that A, should also believe that B.

[-][anonymous]16y00

sigh

Not even close, any of you ;)

The question of whether there are any moral givens or not is analagous to the question of whether there are any mathematical givens (which was covered by E.Yudkowsky in an earlier series). Unfortunately, he doesn't seem to have learned the lesson.

Firstly, in the series on mathematics it was (correctly I think) put forward that even mathematics is not in fact, a priori or axiomatic. 2+2=4 for instance, can be empirically based on the observation that when you have two apples, and you add another two apples, you end up with... (read more)

Geddes, if you can't keep the condescension out of your comments - just present the raw arguments, if you have any - then I'll have to ban you here, too. Just FYI. Also, your comments should be shorter.

I think that Subhan and Obert may represent two sides of a false dichotomy, namely the idea that there's either one absolute morality for all minds, or it's all subjective. But a third possibility exists - that of objective morality, where the results depend on the physical nature of the being in question, but not their whims.

-5Peterdjones11y

@ Ian C. Couldn't Subhan claim that as a restatement of his own position? His notion of wanting clearly encompasses more than mere whims. Perhaps he would say that a certain subset of desires, objectively grounded in the constitution of the mind, count as moral impulses.

Actually, is Subhan meant to be male? Apologies if not.

@Lake - I think Subhan is only about whims. Yes, he sees that values are tied closely to human nature, but only uses that to argue against Obert. What Obert should have pointed out is that he goes from "there is not one true morality" to "there is only preference" without arguing why those are the only two possibilities.

FYI, it's physics that is fundamental. Math is deeper than our theories of physics - it's deeper than all our theories, because it makes up the languages we use to create and express them - but physics itself is deeper than everything.

Similarly, if there are any moral givens, I would agree that they have to result in real empirical differences.
Correct.
Again, we could never percieve moral givens directly (since they are abstract)
Not correct - everything we perceive is equally abstract. There are different kinds of abstractions defined by their inte... (read more)

I gestured at one possible answer to that question. A situation has a moral dimension if it engages moral emotions - which can presumably be listed.

Still doesn't tell us what 'moral' means. We've just changed the category we stick that label on. What defines 'moral emotions'? Is it an arbitrary grouping, or do we use the label to refer to certain properties that things in that grouping possess? People seem to use the term in the latter way - so what properties are they?

Basic scientific methodology - you can't study what you can't produce a provisional definition for. Once you have that, you can learn more about what's defined, but you don't get anywhere without that starting point.

Basic scientific methodology - you can't study what you can't produce a provisional definition for. Once you have that, you can learn more about what's defined, but you don't get anywhere without that starting point.

The first concepts that more less denoted, say, water, may have included things which today we would reject as not water (e.g., possibly clear alcohol), failed to distinguish water from things dissolved in the water, and excluded forms of water (such as steam and ice). The very first definitions of water were probably ostensive definitions (thi... (read more)

Are you willing to accept an ostensive and potentially erroneous definition of morality that may very well be subject to revision as knowledge improves?
That's how knowledge works, Constant. Everything we think we know may turn out to be wrong, and any conclusions will probably end up being revised later.

Are you willing to have a neverending discussion, with everyone talking past each other, and no working definition for the central concept we're supposed to be examining?

Are you willing to have a neverending discussion, with everyone talking past each other, and no working definition for the central concept we're supposed to be examining?

I'm not in charge of the discussion, so it's not a question of what I'm willing to do. I've told you how to get the starting definition you're looking for. As I said: you can start with an ostensive definition by listing examples of evil acts. Then you can find common elements. For example, it might become apparent, after surveying them, that evil acts have in common that they all have vic... (read more)

It seems the ultimate confusion here is that we are talking about instrumental values . . . before agreeing on terminal values . . .

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

denis bider, I would not be surprised to learn that refraining from murder is a terminal value for Eliezer. Eliezer's writings imply that he has hundreds of terminal values: he cannot even enumerate them all.

Defn. "Murder" is killing under particular circumstances, e.g., not by uniformed soldiers during a war, not in self-defense, not by accident.

Great dialog, which I think can be summarized in Nietzsche's aphorism: "Morality is the herd-instinct in the individual."

Actually, I think the dialog could have been a lot shorter if it became clear earlier on that preference (as in morality-as-preference) referred not to individual preference but the "preference of the collective". Which is to say, morality is determined by evolutionary psychology. There are however two assumptions built into the evolutionary psychological explanation of morality which ought to be made explicit.

The fir... (read more)

My comment is not charitable enough towards the CEVists. I ask the moderator to delete it, I will now submit a replacement.

It seems the ultimate confusion here is that we are talking about instrumental values . . . before agreeing on terminal values . . .

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

denis bider, under the CEV plan for singularity, no human has to give an unambiguous definition or enumeration of his or her terminal values before the launch of the seed of the superintelligence. Consequently, those who lean toward th... (read more)

As I said: you can start with an ostensive definition by listing examples of evil acts. Then you can find common elements.

There aren't necessarily any common elements, besides utterly trivial ones. If you look at examples of misspelled words in various languages and examine their individual properties, you won't find what unites them in a category. You have to understand their relationship to the spelling rules in the various languages - rules which themselves are likely to be incompatible and mutually incoherent - to understand what properties make t... (read more)

Quick comment: Terren Suydam's version of "evolutionary psychology" is not the academically accepted one. Conventional academic evolutionary explanations of morality rely on neither group selection nor selection on cultures.

Richard, once we can see how to eliminate math or science views, then it doesn't seem particularly puzzling that people can generate plausible views to consider. The obvious hypothesis is that they generate many views in their mind, apply crude but effective filters, and then only tell others about the few best ones. So of course the views they propose will fare far better than average when tested against consistency and data.

Quick comment: Terren Suydam's version of "evolutionary psychology" is not the academically accepted one. Conventional academic evolutionary explanations of morality rely on neither group selection nor selection on cultures.

Be that as it may, I would have to say that an explanation of morality made strictly in terms of the academically accepted version of evolutionary psychology is not possible. I'm not trying to redefine the term - just saying what else would be necessary to make an explanation of morality on that basis possible.

There aren't necessarily any common elements, besides utterly trivial ones.

Maybe, maybe not. You won't know without looking. You have to start somewhere.

If you look at examples of misspelled words in various languages and examine their individual properties, you won't find what unites them in a category.

But then, what about correctly spelled words? There will be many observable systematic relationships between those. I happen to think you have the analogy backwards. In the good/evil dichotomy, it is the evil acts, not the not-evil acts, which are narrowly ... (read more)

Terren Suydam: "The first is that one has to adopt the group-selection stance."

(Technical jargon nitpick.)

In studying evolutionary biology, "group-selection" has a specific meaning, an individual sacrifices its own fitness in order to improve the group fitness. I.e., individual loss for a group gain. E.g., suppose you have a species that consists of many small family groups. Suppose a mutation produces a self-sacrificing individual in one of the groups. His fitness is slightly lower but his family group fitness is higher. His group tend... (read more)

Nobody told Galileo and Newton what the rules generating the world's behavior were, but they were able to go a long way toward figuring them out.
They started with rigorously-defined phenomena. Galileo was famous for taking careful and exacting measurements - he was able to figure out some of the world's rules because he could notice the relationships between precisely-defined and -measured things.

Nobody gets anywhere, most especially in philosophy, without rigorous definitions of the relevant concepts. You rely on vague, intuitive understandings, and you accomplish nothing.

In studying evolutionary biology, "group-selection" has a specific meaning, an individual sacrifices its own fitness in order to improve the group fitness.

I think it's quite limiting to think strictly in terms of genetics, because there is more than one level of description going on when it comes to selection pressure.

It is interesting to take that step back and view the culture as an individual. The human super-organism (e.g., a tribe, or more generally, a culture) competes with others for resources. It consumes, metabolizes, and excretes, which... (read more)

Terren, I would doubt that changes between cultures are best explained by an evolutionary process--cf. "No Evolutions for Corporations or Nanodevices." There may be a selection effect in that cultures with guns are more likely to persist, but that's different from saying that selection pressures play a really important role in designing the particular features of a culture. So I am given to understand.

There may be a selection effect in that cultures with guns are more likely to persist, but that's different from saying that selection pressures play a really important role in designing the particular features of a culture.

That's what I'm saying - selection pressures are important in determining cultural features, because those features in turn determine a culture's viability. The global-level organization of a culture - including its moral code, political organization, and other important social structures - are key considerations in what makes a culture... (read more)

Terren Suydam: "So genetics is not the whole story, and that's what I mean by group selection."

I use the term "multilevel selection" for what you are describing. I agree it has been important.

E.g., there has been selection between different species. Species with genomes that supported rapid adaptation to changing environments and that supported quick diversification when expanding into new niches spread far and wide. (Beetles have been extremely successful with around 350,000 known species.) Other specie branches died out. The genetic m... (read more)

[-][anonymous]16y00

>Geddes, if you can't keep the condescension out of your comments - just present the raw arguments, if you have any - then I'll have to ban you here, too. Just FYI. Also, your comments should be shorter.

But all I'm asking for is an explanation as to why decision theory works? Perhaps someone like R.Hanson could explain?

After all, I know (admittedly only in very general terms) what the explanation for thermodynamics is (the underlying explanation is in the concepts of mechanics- energy, force etc.). Also, I know (in general), why probability theory wor... (read more)

"If morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

I'm going to play Devil's Advocate for a moment here. Anyone, please feel free to answer, but do not interpret the below arguments as correlating with my set of beliefs.

"A remarkable coincidence? Of course not! If we're supposing that this 'stone tablet' has some influence on the universe - and if it exists, it must exert influence, otherwise we wouldn't have any evidence wherewith to be arguing over whether or not it exists ... (read more)

2hairyfigment13y
By your Devil's logic here, we would expect at least part of human nature to accord with the whole of this 'stone tablet'. I think we could vary the argument to avoid this conclusion. But as written it implies that each 'law' from the 'tablet' has a reflection in human nature, even if perhaps some other part of human nature works against its realization. This implies that there exists some complicated aspect of human nature we could use to define morality which would give us the same answers as the 'stone tablet'.
1Arandur13y
Which sounds like that fuzzily-defined "conscience" thing. So suppose I say that this "Stone tablet" is not a literal tablet, but is rather a set of rules that sufficiently advanced lifeforms will tend to accord to? Is this fundamentally different than the opposite side of the argument?
0hairyfigment13y
Well, that depends. What does "sufficiently advanced" mean? Does this claim have anything to say about Clippy? If it doesn't constrain anticipation there, I suspect no difference exists.
0Arandur13y
Ha! No. I guess I'm using a stricter definition of a "mind" than is used in that post: one that is able to model itself. I recognize the utility of such a generalized definition of intelligence, but I'm talking about a subclass of said intelligences.
3hairyfigment13y
Er, why couldn't Clippy model itself? Surely you don't mean that you think Clippy would change its end-goals if it did so (for what reason?)
6Arandur13y
... Just to check: we're talking about Microsoft Office's Clippy, right?
5Alicorn13y
Not likely.
1Arandur13y
Oh dear; how embarrassing. Let me try my argument again from the top, then.
1orthonormal13y
Actually, this is what we're really talking about, not MS Word constructs or LW roleplayers.

« Then you believe in universally compelling arguments processed by a ghost in the machine. For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value -1. »

That's true but that doesn't say anything about the sustainability of that given mind design (the possibility for the mind design to survive, either by having the individual to survive, or to create new individuals with similar mind designs).

A mind design that would value its own death ... (read more)

[-][anonymous]12y-10

It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow.

Having an authority to follow might actually be that mind's one good thing. Maybe it really likes having such authority beyond itself.

While humans obviously don't consider that their only good this and there is human variation (I don't think everyone values it, I'm only certain at least a few do), it seems pretty clear to me that one of our good things, as in stuff we find worth bothering... (read more)

It is sometimes argued that happiness is good and suffering is bad. (This is tentatively my own view, but explaining the meaning of "good" and "bad," defending its truth, and expanding the view to account for the additional categories of "right" and "wrong" is beyond the scope of this comment.)

If this is true, then depending on what kind of truth it is, it may also be true in all possible worlds--and a fortiori, on all possible planets in this universe. Furthermore, if it is true on all possible planets that happines... (read more)

0themusicgod17y
Similarly, an even more defensible position might be Buddhist one, or that happiness is transitory and mostly a construction of the mind, and virtually always attached to suffering, but suffering is real and worth minimizing.
By changing a mind, you can change what it prefers; you can even change what it believes to be right; but you cannot change what is right.  Anything you talk about, that can be changed in this way, is not 'right-ness'.

If the characters were real people, I'd say here Obert is "right" while having a wrong justification. Just extrapolate the evolutionary origins of moral intuitions into any society in approximate technological stasis. "Rightness" is how the evolutionarily stable strategy feels like from the inside, an... (read more)