Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Moral Error and Moral Disagreement

14 Post author: Eliezer_Yudkowsky 10 August 2008 11:32PM

Followup toInseparably Right, Sorting Pebbles Into Correct Heaps

Richard Chappell, a pro, writes:

"When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I don't see how Eliezer can accommodate this. On his account, what Bob asserted is true iff abortion is prohibited by the morality_Bob norms. How can Sally disagree? There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob...

Since there is moral disagreement, whatever Eliezer purports to be analysing here, it is not morality."

The phenomena of moral disagreement, moral error, and moral progress, on terminal values, are the primary drivers behind my metaethics.  Think of how simple Friendly AI would be if there were no moral disagreements, moral errors, or moral progress!

Richard claims, "There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob."

We may not suppose, and there is disputing.  Bob does not have direct, unmediated, veridical access to the output of his own morality.

I tried to describe morality as a "computation".  In retrospect, I don't think this is functioning as the Word of Power that I thought I was emitting.

Let us read, for "computation", "idealized abstract dynamic"—maybe that will be a more comfortable label to apply to morality.

Even so, I would have thought it obvious that computations may be the subjects of mystery and error.  Maybe it's not as obvious outside computer science?

Disagreement has two prerequisites: the possibility of agreement and the possibility of error.  For two people to agree on something, there must be something they are agreeing about, a referent held in common.  And it must be possible for an "error" to take place, a conflict between "P" in the map and not-P in the territory.  Where these two prerequisites are present, Sally can say to Bob:  "That thing we were just both talking about—you are in error about it."

Richard's objection would seem in the first place to rule out the possibility of moral error, from which he derives the impossibility of moral agreement.

So: does my metaethics rule out moral error?  Is there no disputing that abortion is indeed prohibited by morality_Bob?

This is such a strange idea that I find myself wondering what the heck Richard could be thinking.  My best guess is that Richard, perhaps having not read all the posts in this sequence, is taking my notion of morality_Bob to refer to a flat, static list of valuations explicitly asserted by Bob.  "Abortion is wrong" would be on Bob's list, and there would be no disputing that.

But on the contrary, I conceive of morality_Bob as something that unfolds into Bob's morality—like the way one can describe in 6 states and 2 symbols a Turing machine that will write 4.640 × 101439 1s to its tape before halting.

So morality_Bob refers to a compact folded specification, and not a flat list of outputs.  But still, how could Bob be wrong about the output of his own morality?

In manifold obvious and non-obvious ways:

Bob could be empirically mistaken about the state of fetuses, perhaps believing fetuses to be aware of the outside world.  (Correcting this might change Bob's instrumental values but not terminal values.)

Bob could have formed his beliefs about what constituted "personhood" in the presence of confusion about the nature of consciousness, so that if Bob were fully informed about consciousness, Bob would not have been tempted to talk about "the beginning of life" or "the human kind" in order to define personhood.  (This changes Bob's expressed terminal values; afterward he will state different general rules about what sort of physical things are ends in themselves.)

So those are the obvious moral errors—instrumental errors driven by empirical mistakes; and erroneous generalizations about terminal values, driven by failure to consider moral arguments that are valid but hard to find in the search space.

Then there are less obvious sources of moral error:  Bob could have a list of mind-influencing considerations that he considers morally valid, and a list of other mind-influencing considerations that Bob considers morally invalid.  Maybe Bob was raised a Christian and now considers that cultural influence to be invalid.  But, unknown to Bob, when he weighs up his values for and against abortion, the influence of his Christian upbringing comes in and distorts his summing of value-weights.  So Bob believes that the output of his current validated moral beliefs is to prohibit abortion, but actually this is a leftover of his childhood and not the output of those beliefs at all.

(Note that Robin Hanson and I seem to disagree, in a case like this, as to exactly what degree we should take Bob's word about what his morals are.)

Or Bob could believe that the word of God determines moral truth and that God has prohibited abortion in the Bible.  Then Bob is making metaethical mistakes, causing his mind to malfunction in a highly general way, and add moral generalizations to his belief pool, which he would not do if veridical knowledge of the universe destroyed his current and incoherent metaethics.

Now let us turn to the disagreement between Sally and Bob.

You could suggest that Sally is saying to Bob, "Abortion is allowed by morality_Bob", but that seems a bit oversimplified; it is not psychologically or morally realistic.

If Sally and Bob were unrealistically sophisticated, they might describe their dispute as follows:

Bob:  "Abortion is wrong."

Sally:  "Do you think that this is something of which most humans ought to be persuadable?"

Bob:  "Yes, I do.  Do you think abortion is right?"

Sally:  "Yes, I do.  And I don't think that's because I'm a psychopath by common human standards.  I think most humans would come to agree with me, if they knew the facts I knew, and heard the same moral arguments I've heard."

Bob:  "I think, then, that we must have a moral disagreement: since we both believe ourselves to be a shared moral frame of reference on this issue, and yet our moral intuitions say different things to us."

Sally:  "Well, it is not logically necessary that we have a genuine disagreement.  We might be mistaken in believing ourselves to mean the same thing by the words right and wrong, since neither of us can introspectively report our own moral reference frames or unfold them fully."

Bob:  "But if the meaning is similar up to the third decimal place, or sufficiently similar in some respects that it ought to be delivering similar answers on this particular issue, then, even if our moralities are not in-principle identical, I would not hesitate to invoke the intuitions for transpersonal morality."

Sally:  "I agree.  Until proven otherwise, I am inclined to talk about this question as if it is the same question unto us."

Bob:  "So I say 'Abortion is wrong' without further qualification or specialization on what wrong means unto me."

Sally:  "And I think that abortion is right.  We have a disagreement, then, and at least one of us must be mistaken."

Bob:  "Unless we're actually choosing differently because of in-principle unresolvable differences in our moral frame of reference, as if one of us were a paperclip maximizer.  In that case, we would be mutually mistaken in our belief that when we talk about doing what is right, we mean the same thing by right.  We would agree that we have a disagreement, but we would both be wrong."

Now, this is not exactly what most people are explicitly thinking when they engage in a moral dispute—but it is how I would cash out and naturalize their intuitions about transpersonal morality.

Richard also says, "Since there is moral disagreement..."  This seems like a prime case of what I call naive philosophical realism—the belief that philosophical intuitions are direct unmediated veridical passports to philosophical truth.

It so happens that I agree that there is such a thing as moral disagreement.  Tomorrow I will endeavor to justify, in fuller detail, how this statement can possibly make sense in a reductionistic natural universe.  So I am not disputing this particular proposition.  But I note, in passing, that Richard cannot justifiably assert the existence of moral disagreement as an irrefutable premise for discussion, though he could consider it as an apparent datum.  You cannot take as irrefutable premises, things that you have not explained exactly; for then what is it that is certain to be true?

I cannot help but note the resemblance to Richard's assumption that "there's no disputing" that abortion is indeed prohibited by morality_Bob—the assumption that Bob has direct veridical unmediated access to the final unfolded output of his own morality.

Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob, and allowed by morality_Sally, there being at least two possible minds for whom this would be true.  Then the two minds might be mistaken about believing themselves to disagree.  Actually they would simply be directed by different algorithms.

You cannot have a disagreement about which algorithm should direct your actions, without first having the same meaning of should—and no matter how you try to phrase this in terms of "what ought to direct your actions" or "right actions" or "correct heaps of pebbles", in the end you will be left with the empirical fact that it is possible to construct minds directed by any coherent utility function.

When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes.  You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about—unless you cover "disagreement" to include differences where two agents have nothing to say to each other.

But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so.  Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths.  If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it.  Now, perhaps some psychopaths would not be persuadable in-principle to take the pill that would, by our standards, "fix" them.  But I note the possibility to emphasize what an extreme statement it is to say of someone:

"We have nothing to argue about, we are only different optimization processes."

That should be reserved for paperclip maximizers, not used against humans whose arguments you don't like.

 

Part of The Metaethics Sequence

Next post: "Abstracted Idealized Dynamics"

Previous post: "Sorting Pebbles Into Correct Heaps"

Comments (125)

Sort By: Old
Comment author: Nick_Tarleton 10 August 2008 11:59:42PM 6 points [-]

Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.

Comment author: benelliott 09 June 2011 01:16:22PM 1 point [-]

I believe that zero out of 6.5 billion humans are unicorns.

The big number does not create an implausibility all on its own, there may be stable psychopaths, but I wouldn't be very surprised if there weren't.

Comment author: Arandur 12 August 2011 12:22:44PM 4 points [-]

Far less likely to suppose that 1 out of 6.5 billion humans is a stable psychopath.

Comment author: steven 11 August 2008 12:22:06AM 2 points [-]

I wonder if the distinction between 1) something implementing the same dynamic as a typical human but mistaken about what it says and 2) something implementing a completely different dynamic and not mistaken about anything, is the same as the distinction people normally make between 1) immoral and 2) amoral.

Comment author: Jadagul 11 August 2008 01:14:27AM 1 point [-]

Steven: quite possibly related. I don't think they're exactly the same (the classic comic book/high fantasy "I'm evil and I know it" villain fits A2, but I'd describe him as amoral), but it's an interesting parallel.

Eliezer: I'm coming more and more to the conclusion that our main area of disagreement is our willingness to believe that someone who disagrees with us really "embodies a different optimization process." There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processes; while I believe mine to be the best I've found, I remain aware that if I held any of the others I would believe exactly the same thing. And that I would have no way of convincing the anti-Occam intelligence that Occam's Razor was a good heuristic, or of convincing the psychopath who really doesn't care about other people that he 'ought' to. So I hesitate to say that I'm right in any objective sense, since I'm not sure exactly what standard I'm pointing to when I say 'objective.'

And I've had extended moral conversations with a few different people that led to us, eventually, concluding that our premises were so radically different that we really couldn't have a sensible moral conversation. (to wit: I think my highest goal in life is to make myself happy. Because I'm not a sociopath making myself happy tends to involve having friends and making them happy. But the ultimate goal is me. Makes it hard to talk to someone who actually believes in some form of altruism).

Comment author: TGGP4 11 August 2008 01:48:22AM 0 points [-]

If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all. A General AI should be able to convince it just about anything, right?

Comment author: Eliezer_Yudkowsky 11 August 2008 02:02:42AM 6 points [-]

Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.

Heck, what about babies? What do they want, and would they be complicated enough to want anything different if they knew more and thought faster?

There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processes; while I believe mine to be the best I've found, I remain aware that if I held any of the others I would believe exactly the same thing.

You would not believe exactly the same thing. If you held one of the others, you would believe that your new system was frooter than any of the others, where "frooter" is not at all the same thing as "better". And you would be correct.

If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all. A General AI should be able to convince it just about anything, right?

If you make matters that complicated to begin with, i.e., we're not discussing metaethics for human usage anymore, then you should construe entailment / extrapolation / unfolding in more robust ways than "anything a superintelligence can convince you of". E.g. CEV describes a form of entailment.

As for what a person's morality is, surely you extrapolate it at least a little beyond their instantaneous beliefs. Would you agree that many people would morally disapprove of being shot by you, even if the actual thought has never crossed their mind and they don't know you exist?

Comment author: conchis 11 August 2008 02:06:59AM 0 points [-]

I must be starting to get it. That unpacked in exactly the way I expected.

On the other hand, this:

If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all.

makes no sense to me at all.

Comment author: Sebastian_Hagen2 11 August 2008 02:24:59AM 0 points [-]

I think my highest goal in life is to make myself happy. Because I'm not a sociopath making myself happy tends to involve having friends and making them happy. But the ultimate goal is me.

If you had a chance to take a pill which would cause you to stop caring about your friends by permanently maxing out that part of your hapiness function regardless of whether you had any friends, would you take it?
Do non-psychopaths that given the chance would self-modify into psychopaths fall into the same moral reference frame as stable psychopaths?

Comment author: jsalvati 11 August 2008 02:54:24AM 1 point [-]

Sebastian Hagen:

My intuition is that a good deal of people would take the psychopath pill. At least if the social consequences were minimal, which is besides the point.

Comment author: Hopefully_Anonymous 11 August 2008 03:16:44AM 2 points [-]

Frame it defensively rather than offensively and a whole heck of a lot of people would take that pill. Of course some of us would also take the pill that negates the effects of our friends taking the first pill, hehehe.

Comment author: Hopefully_Anonymous 11 August 2008 03:17:50AM 0 points [-]

Weird, jsalvati is not my sock puppet, but the 11:16pm post above is mine.

Comment author: Eliezer_Yudkowsky 11 August 2008 03:21:42AM 0 points [-]

Fixed. How very odd. (And for the record, jsalvati's IP is not HA's.)

Comment author: Richard4 11 August 2008 03:32:24AM 1 point [-]

"Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob..."

That's right. (I didn't mean to suggest that there's never any disputing what someone's moral commitments are; just that this wasn't supposed to be in dispute in the particular case I was imagining.) I take it that Sally and Bob could disagree even so, and not merely be talking past each other, even if one or both of them was impervious to rational argument. It is at least a significant cost of your theory that it denies this datum. (It doesn't have to be 'irrefutable' to nonetheless be a hefty bullet to bite!)

I like your account of everyday moral disagreement, but would take it a step further: it is no mere accident that your Sally and Bob expect to be able to persuade the other. Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents.

It's worth emphasizing, though, that humans are not fully reasonable. Some are even irrevocably unreasonable, incapable of rationally updating (some of) their beliefs. So while, I claim, we all aspire to the morality_Objective norms, I doubt there's any empirically specifiable procedure that could ensure our explicit affirmation of those norms (let alone their unfolded implications). Bob may stubbornly insist that abortion is wrong, and this may conflict with other claims he makes, but there's simply no way (short of brain surgery) to shake him from his illogic. What then? I say he's mistaken, even though he can't be brought to recognize this himself. It's not clear to me whether you can say this, since I'm not sure exactly what your 'extrapolation' procedure for defining morality_Bob is. But if it's based on any simple empirical facts about what Bob would believe if we told him various facts and arguments, then it doesn't look like you'll be able to correct for the moral errors that result from sheer irrationality, or imperviousness to argument.

Could you say a little more about exactly which empirical facts serve to define morality_Bob?

Comment author: Mike_Blume 11 August 2008 04:25:08AM 1 point [-]

I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you.

Do you imagine that there are facts about the brains of cattle which we could both learn - facts drawn from fMRI scans, or from behavioral science experiments, perhaps - which would bring us into agreement on the issue?

Comment author: Unknowns 01 February 2010 08:15:46AM 1 point [-]

No. I agree with those facts and think that eating steak is a good idea anyway.

Comment author: Eliezer_Yudkowsky 01 February 2010 08:33:36AM 0 points [-]

I'm curious: Would you eat Yoda?

BTW, if you're the original Unknown, please see Open Thread for a query about our outstanding bet.

Comment author: Unknowns 01 February 2010 09:58:22AM 0 points [-]

I would eat sentient creatures but not intelligent creatures. And even in the case of sentient non-intelligent ones it depends on what I'm used to.

Comment author: Unknowns 01 February 2010 10:05:39AM 1 point [-]

By the way, is there any way to get back my original username? I didn't know how to do it when things were moved over from OB.

Comment author: Jadagul 11 August 2008 06:22:54AM 0 points [-]

Eliezer: for 'better' vs 'frooter,' of course you're right. I just would have phrased it differently; I've been known to claim that the word 'better' is completely meaningless unless you (are able to) follow it with "better at or for something." So of course, Jadagul_real would say that his worldview is better for fulfilling his values. And Jadagul_hypothetical would say that his worldview is better for achieving his values. And both would (potentially) be correct. (or potentially wrong. I never claimed to be infallible, either in reality or in hypothesis). But phrasing issues aside, I do believe that I think this happens more often than you think it happens.

Sebastian Hagen: That's actually a very good question. So a few answers. First is that I tend to go back and forth on whether by 'happiness' I mean something akin to "net stimulation of pleasure centers in brain," or to "achievement of total package of values" (at which point the statement nears tautology, but I think doesn't actually fall into it). But my moral code does include such statements as "you have no fundamental obligation to help other people." I help people because I like to. So I lean towards formulation 1; but I'm not altogether certain that's what I really mean.

Second is that your question, about the sociopath pill, is genuinely difficult for me. It reminds me of Nozick's experience machine thought experiment. But I know that I keep getting short-circuited by statements like, "but I'd be miserable if I were a sociopath," which is of course false by hypothesis. I think my final answer is that I'm such a social person and take such pleasure in people that were I to become a sociopath I would necessarily be someone else. That person wouldn't be me. And while I care about whether I'm happy, I don't know that I care about whether he is.

Of course, this all could be "I know the answer and now let me justify it." On the other hand, the point of the exercise is to figure out what my moral intuitions are...

Comment author: Tim_Tyler 11 August 2008 06:50:20AM 0 points [-]

Re: "We have nothing to argue about, we are only different optimization processes." That seems to apply in the case when a man wants to rescue copies of his genes from eternal oblivion - by convincing his mate not to abort his prospective offspring. Of course, not many would actually say that under those circumstances.

Comment author: Virge2 11 August 2008 06:51:29AM 1 point [-]

Eliezer: "But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it."

How sure are you that most human moral disagreements are attributable to - lack of veridical information, or - lack of ability/tools to work through that information, or - defects? You talk freely about psychopaths and non-psychopaths as though these were distinct categories of non-defective and defective humans. I know you know this is not so. The arguments about psychological unity of humankind only extend so far. e.g., would you be prepared to tell a homosexual that, if they were fully informed, they would decide to take a pill to change their orientation? There are demonstrable differences where our fellow humans really are "different optimization processes". Why should we ignore the spread of differences in moral computations?

I've been enjoying your OB posts and your thought experiments are powerful, but I'm curious as to the empirical data that have led you to update your beliefs so strongly in favour of psychological unity and so strongly against differences in computation. Your arguments that mention psychopaths smack a little of a "no true Scotsman" definition of human morality.

Comment author: John_T._Kennedy 11 August 2008 07:02:52AM 0 points [-]

Mike,

"I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you."

Accepting your propositions for the sake of argument, I still find that eating steak seems reasonable.

Comment author: Carl_Shulman 11 August 2008 07:06:32AM 2 points [-]

"Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents."

Richard,

If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'

Comment author: Tim_Tyler 11 August 2008 07:10:29AM 0 points [-]

Other moral issues where there are a gender differences include: "should prostitution be legalised" and "should there be tighter regulation of pornography".

Again, it seems that part of the effect is due to people's idea of what is right being influenced by their own personal role - i.e. the "different optimization processes" effect.

Gender is the most obvious source of such issues, but I'm sure you can find them in other areas of life. Race politics, for instance.

Comment author: TGGP4 11 August 2008 07:35:14AM 1 point [-]

People in general do not want to be shot. The person doing the shooting, the lethal weapon being fired, the location in which the shooting occurs and the time of day are all pretty much irrelevant. You can ask people if they want to be shot and they'll say no, without even specifying those details. That seems a very different case from Bob, who is considering a moral proposition and outright rejecting it.

Comment author: Simon_M 11 August 2008 07:43:00AM 0 points [-]

Given all of Bacon's idols of the Mind can you ever know definitely if there is an error in your own reasoning, let alone the other persons?

You cannot rely on your moral intuition, nor the cultural norms of your time, nor academic authority, nor your internal reasoning or ability to know the soundness of your argument.

Socialization, severe biases, faulty reasoning can all make you think you are ‘correct’, but can leave you with the incorrect impression of the ‘correctness’ of your thinking. & even if presented with all the correct or relevant information some people still make these errors, so if they can so could you.

Comment author: Will_Pearson 11 August 2008 11:56:40AM 0 points [-]

You have been priming people to think in terms of functions. (Pure) Functions do not change. They map an input to an output, and can be substituted by a list, e.g. a function that tests for primality can can be performed by an (infinitely) long list.

You may want to describe impure functions for people without a functional programming background if you want to keep terminology like morality_john().

Comment author: Schizo 11 August 2008 12:27:54PM 0 points [-]

Virge's point seems particularly important, and hopefully Eliezer can address it.

Comment author: Zubon 11 August 2008 01:27:00PM 0 points [-]

I find Roko on-point. The psychological unity of humankind is important, but it can be over-stated. While human beings may occupy a very small area in the space of all possible minds, it is still an area and not a single point. When we shut up and multiply by sufficiently large numbers, very small differences in the starting point are very meaningful. If we are talking about a difference valued at 0.0000001% of a human life, and you extrapolate it over a billion lives, we are talking about life and death matters. Successful AI will affect more than a billion lives.

If you are the advocate of 3^^^3 specks, you must take seriously the notion that 0.0000001% differences in morality will be meaningful when multiplied over all minds into the indefinite future.

Comment author: Eliezer_Yudkowsky 11 August 2008 01:39:37PM 4 points [-]

Virge:

How sure are you that most human moral disagreements are attributable to - lack of veridical information, or - lack of ability/tools to work through that information, or - defects? You talk freely about psychopaths and non-psychopaths as though these were distinct categories of non-defective and defective humans. I know you know this is not so. The arguments about psychological unity of humankind only extend so far. e.g., would you be prepared to tell a homosexual that, if they were fully informed, they would decide to take a pill to change their orientation?

I don't claim to be sure at all. It does seem to me that most modern humans are very much creatures of their own time; they don't consider that future moral progress could be as radical as the change between their own time and Archimedes's; they think of themselves as the wise, the educated, the modern, not as the savage barbarian children the Future will very likely regard us as. It also seems to me that people fail to systematically distinguish between terminal and instrumental disputes, and that they demonize their enemies (which is correspondence bias). The basic ev-bio necessity behind the psychological unity of human brains is not widely understood.

And even more importantly, the portion of our values that we regard as transpersonal, the portion we would intervene to enforce against others, is not all of our values; it's not going to include a taste for pepperoni pizza, or in my case, it's not going to include a notion of heterosexuality or homosexuality.

If there are distinct categories of human transpersonal values, I would expect them to look like "male and female babies", "male children", "male adults", "female children", "female adults", "neurological damage 1", "neurological damage 2", not "Muslims vs. Christians!"

Roko:

If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it.

- no, you wouldn't. The only reason that you are now saying that you would take it is that you currently have the ability to care about other people.

I said "damaged" not "missing". The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.

(Albeit if the damage was in any way entwined with my ability to do Singularity work, I would delay taking the pill.)

Let me spell it out. Every human mind comes with an evolved set of "yuck" factors (and their opposite, which I might call "yum" factors?). This is the "psychological unity of humankind". Unfortunately, these cover only those situations which we were likely to run into in our EEA. Abortion probably did not exist in our EEA: so people have to compare it to something that did. There are two ways to do this - either you think of it as being just like helping a fellow member of your tribe, and become pro-abortion, or you think of it as being infanticide and become anti abortion. Beyond these "yuck" factors, there is no further unity to the moral views of humankind.

The question is how much of this disagreement would persist if the disputants had full veridical knowledge of everything that goes on inside the developing fetus and full veridical knowledge of a naturalistic universe. Given the same "yum" and "yuck" factors, why would they be differently targeted? If there are no interpersonally compelling reasons to target them one way or the other, how would a group of fully informed minds come to believe that this was a proper issue of transpersonal morality?

By the way, quite a number of hunter-gatherer tribes practice infanticide as a form of birth control.

Zubon:

If we are talking about a difference valued at 0.0000001% of a human life, and you extrapolate it over a billion lives, we are talking about life and death matters. Successful AI will affect more than a billion lives.

If I have a value judgment that would not be interpersonally compelling to a supermajority of humankind even if they were fully informed, then it is proper for me to personally fight for and advocate that value judgment, but not proper for me to preemptively build an AI that enforces that value judgment upon the rest of humanity. The notion of CEV reflects this statement of morality I have just made.

Comment author: steven 11 August 2008 02:04:46PM 0 points [-]

I must be missing something -- why would you advocate something that you know you can't justify to anyone else?

Comment author: Zubon 11 August 2008 02:09:19PM 5 points [-]

I said "damaged" not "missing". The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.

Does that work in the other direction? The notion is that you are your current self, but one day I inform you that, relative to other humans, your ability to care about others is damaged in the sense that it is hyperactive. Do you want a pill that will fix the damage and change your values so that you value humanity less (putting you in a less lonely volume of mindspace)?

(And if you do not think that there is a meaningful chance that you should value humanity less, doesn't that mean you are most likely undervaluing humanity right now, so you should update and start valuing humanity more? This is a general you, for everyone: if you do not think you should be a bit more sociopathic, what are the odds you have exactly the right amounts of empathy and altruism?)

Comment author: Sebastian_Hagen2 11 August 2008 03:04:03PM 0 points [-]

Jadagul:

But my moral code does include such statements as "you have no fundamental obligation to help other people." I help people because I like to.

While I consider myself an altruist in principle (I have serious akrasia problems in practice), I do agree with this statement. Altruists don't have any obligation to help people, it just often makes sense for them to do so; sometimes it doesn't, and then the proper thing for them is not to do it.

Roko:

In the modern world, people have to make moral choices using their general intelligence, because there aren't enough "yuck" and "yum" factors around to give guidance on every question. As such, we shouldn't expect much more moral agreement from humans than from rational (or approximately rational) AIs.

There might not be enough "yuck" and "yum" factors around to offer direct guidance on every question, but they're still the basis for abstract rational reasoning. Do you think "paperclip optimizer"-type AIs are impossible? If so, why? There's nothing incoherent about a "maximize the number of paperclips over time" optimization criterion; if anything, it's a lot simpler than those in use by humans.

Eliezer Yudkowsky:

If I have a value judgment that would not be interpersonally compelling to a supermajority of humankind even if they were fully informed, then it is proper for me to personally fight for and advocate that value judgment, but not proper for me to preemptively build an AI that enforces that value judgment upon the rest of humanity.

I don't understand this at all. How is building a superintelligent AI not just a (highly effective, if you do it right) special method of personally fighting for your value judgement? Are you saying it's ok to fight for it, as long as you don't do it too effectively?

Comment author: Sebastian_Hagen2 11 August 2008 03:06:36PM 0 points [-]

Quick correction: s/abstract rational reasoning/abstract moral reasoning/

Comment author: Larry_D'Anna 11 August 2008 05:04:30PM 0 points [-]

Will Pearson: Why not just treat them as pure functions in the State monad?

Comment author: Tim_Tyler 11 August 2008 05:40:36PM 1 point [-]

Re: If there are distinct categories of human transpersonal values, I would expect them to look like "male and female babies", "male children", "male adults", "female children", "female adults", "neurological damage 1", "neurological damage 2", not "Muslims vs. Christians!"

That seems like the position you would get if you thought that cultural evolution could not affect people's values.

Comment author: Richard4 11 August 2008 06:46:32PM 0 points [-]

Carl - "If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'"

I don't mean to define 'fully reasonable' at all (though it is meant to be minimally ad hoc or gerrymandered). I take this normative notion as a conceptual primitive, and then hypothesize that it entails a certain set of moral norms. They're probably not even my norms (in any way Eliezer could accommodate), since I'm presumably not fully reasonable myself. But they're what I'm trying to aim for, even if I don't always grasp them correctly.

This may sound mysterious and troublingly ungrounded to you. Yet you use terms like 'superintelligent' and 'superhuman understanding', which are no less normative than my 'reasonable'. I think that reasonableness is a component of intelligence and (certainly) understanding, so I don't see how these terms could properly apply to a pencil maximizer. Maybe you simply mean that it is a pencil maximizer that is instrumentally rational and perfectly proficient at Bayesian updating. But that's not to say it's intelligent. It might, for example, be a counterinductivist (didn't someone mention anti-Occamists up-thread?), with completely wacky priors. I take it as a datum that this is simply unreasonable -- there are other norms, besides conditionalization and instrumental rationality, which govern 'intelligent' or good thinking.

So I say there are brute, unanalysable facts about what's reasonable. The buck's gotta stop somewhere. I don't see that any alternative theory does better than this one.

Comment author: Z._M._Davis 11 August 2008 06:59:14PM 1 point [-]

"If there are distinct categories of human transpersonal values, I would expect them to look like [...] 'male adults', [...] 'female adults', [...] not 'Muslims vs. Christians!'"

Really? In the ways that are truly important, don't you think you have more in common with Natasha Vita-More than Osama bin Laden?

Comment author: steven 11 August 2008 07:37:12PM 0 points [-]

ZM, he means after volition-extrapolating.

Comment author: Z._M._Davis 11 August 2008 07:59:48PM 2 points [-]

Steven, even so, I think the basic question stands. Why should cultural differences and within-sex individual differences wash out of the CEV?

Comment author: steven 11 August 2008 08:18:44PM 0 points [-]

Supposedly genetics allows for people of different ages or sexes to have different mental machinery, whereas individual genetic differences just represent low-complexity differences in tweaking. I'm not sure why that makes Eliezer's point though, if the aforementioned differences in tweaking mean different complex machinery gets activated. Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in.

Comment author: Z._M._Davis 11 August 2008 08:39:28PM 2 points [-]

Steven: "Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in."

I suspect a category error here hinging around personal identity. We say "if I had grown up in a different culture ..." when I think we mean "if the baby that grew into me had grown up in a different culture..." If the baby that grew into me had grown up in a radically different culture, I don't think ve'd be me in any meaningful sense, although of course there would be many similarities due to evo-psych and genetics. Whereas I would likely consider a hypothetical member of the opposite sex who shared my ideas and concerns to be me, even if ve was much better or worse than this me at mentally rotating 3D objects, or finding things in the refrigerator.

Comment author: Tim_Tyler 11 August 2008 10:37:04PM 0 points [-]

Re: Roko's "more reasonable position":

Human psychological unity, to the extent that it exists, includes brain speed, brain degree-of-parallelism, brain storage capacity, brain reliability - and other limitations that have little to do with human goals.

Comment author: Bernard_Guerrero2 11 August 2008 10:41:39PM 0 points [-]

If you had a chance to take a pill which would cause you to stop caring about your friends by permanently maxing out that part of your hapiness function regardless of whether you had any friends, would you take it?

I'm not sure this proves anything. I'll take Jadagul's description as my own. I'm maximizing my happiness, but my non-sociopath status means that I like having friends/loved ones, so maximizing my own happiness entails caring about them to some degree or another. Under my existing moral code I wouldn't take the pill, but that's because it will in the future entail behavior that I would currently, as I am, find obnoxious.

In theory, taking the pill will allow me to be happier by obviating the need to worry about anybody else (along with the resources that entails.) But you can't get there from here. I already know what the pill will make me later, and the prospect of taking it makes me unhappy now. Taking the pill would be a viable option only if the behavior the pill would induce could somehow be made acceptable to current, non-sociopath me.

Comment author: Doug_S. 12 August 2008 12:32:17AM 2 points [-]

I, too, wonder if the "psychological unity of humankind" has been a bit overstated. All [insert brand and model here] computers have identical hardware, but you can install different software on them. We're running different

Consider the case of a "something maximizer". It's given an object, and then maximizes the number of copies of that object.

You give one something maximizer a paperclip, and it becomes a paperclip maximizer. You give another a pencil, and that one becomes a pencil maximizer.

There's no particular reason to expect the paperclip maximizer and the pencil maximizer to agree on values, even though they are both something maximizers, implemented on identical hardware. The extrapolated volition of a something-maximizer is not well-defined.

There are probably blank spaces in human Eliezer::morality that are written on by experience, and such writing may well be irrevocable, or mostly irrevocable. If humans' terminal values are, in fact, contingent on experiences, then you could have two people disagreeing on the same level that the two something maximizers do.

As a practical matter, people generally do seem to have similar sets of terminal values, but different people rank the values differently. Consider the case of "honor". Is it better to die with honor or to live on in disgrace? People raised in different cultures will give different answers to this question.

Comment author: Virge2 12 August 2008 04:38:30AM 4 points [-]

Eliezer: "The basic ev-bio necessity behind the psychological unity of human brains is not widely understood."

I agree. And I think you've over-emphasized the unity and ignored evidence of diversity, explaining it away as defects.

Eliezer: "And even more importantly, the portion of our values that we regard as transpersonal, the portion we would intervene to enforce against others, is not all of our values; it's not going to include a taste for pepperoni pizza, or in my case, it's not going to include a notion of heterosexuality or homosexuality."

I think I failed to make my point clearly on the idea of a sexual orientation pill. I didn't want to present homosexuality as a moral issue for your judgment, but as an example of the psychological non-unity of human brains. Many people have made the mistake of assuming that heterosexuality is "normal" and that homosexuals need re-education or "fixing" (defect removal). I hold that sexuality is distributed over a spectrum. The modes of that distribution represent heterosexual males and females--an evolutionary stable pattern. The other regions of that distribution remain non-zero despite selection pressure.

Clearly we do not have a psychological unity of human sexual preference. People with various levels of homosexual preference are not merely defective or badly educated/informed.

Sexuality is a complex feature arising in a complex brain. Because of the required mutual compatability of brain-construction genetics, we can be sure that we all have extremely similar machinery, but our sexual dimorphism requires that a single set of genes can code flexibly for either male or female. Since that flexibility doesn't implement a pure binary male/female switch, we find various in-between states in both physical and mental machinery. The selection pressure from sexual dimorphism means we should expect far more non-unity of sexual preference than in other areas of our psychology.

But the fact that our genes can code for that level of flexibility, yet still remain biologically compatible, tells us that there are likely to be many other computations that could exhibit a superficial unity, but with a broad spread. The spread of propensities to submit to authority and override empathy observed in the Stanford Prison experiment gives good reason to question the supposed unity. (Yes I know you could shoehorn that diversity into your model as a effect of lack of moral training.)

Now let's reconsider psychopathy or the broader diagnosis of antisocial personality disorder. What should we do with those humans whose combination of narcissism and poor social cognition is beyond some particular limit? Lock them up, or elect them to govern? From my limited understanding of the subject, it seems that the "condition" is considered untreatable.

It's an easy path to stick to the psychological unity of humans and declare those in the tails of the distribution to be defective. But is it statistically justified? Does your unity model actually fit the data or just give a better model than the tabula rasa model that Tooby and Cosmides reacted against?

That's why the idea of some idealised human moral computation, that everyone would agree to if they knew enough and weren't defective, seems like question begging. That's why I was asking for the empirical data that have led you to update your beliefs. Then I could update mine from them and maybe we'd be in agreement.

I'm open to the idea that we can identify some best-fit human morality--a compromise that minimizes the distance/disagreement to our population of (veridically informed) humans. That seems to me to be the best we can do.

Comment author: roko3 12 August 2008 12:25:38PM 0 points [-]

virge makes a very good point here. The human mind is probably rather flexible in terms of it's ethical views; I suspect that Eli is overplaying our psychological unity.

Comment author: Eliezer_Yudkowsky 12 August 2008 12:37:35PM 5 points [-]

I think that views are being attributed to me that I do not possess - perhaps on the Gricean notion that if someone attacks me for holding these views, I ought to hold them; but on a blog like this one, that leaves you prey to everyone else's misunderstanding.

I do not assert that all humans end up in the same moral frame of reference (with regard to any particular extrapolation method). I do think that psychological unity is typically underestimated, and I have a hard time taking modern culture at face value (we're the ancient Greeks, guys, not a finished product) - read Tooby and Cosmides's "The Psychological Foundations of Culture" to get a picture of where I'm coming from.

But if you read "Coherent Extrapolated Volition" you'll see that it's specifically designed to handle, among other problems, the problem of, "What if we don't all want the same thing?" What then can an AI programmer do that does not constitute being a jerk? That was my attempt to answer.

Michael Vassar has tried to convince me that I should be more worried about a majority of humanity being "selfish bastards" as defined in CEV. I have a couple of thoughts on this problem, may post on them later.

Comment author: steven 12 August 2008 01:18:38PM 0 points [-]

"Being a jerk" here means "being a jerk according to other people's notion of morality, but not according to my own notion of morality", right?

I sort of take offense to "we're the ancient Greeks"; I make sure to disagree with Western morality whenever it's wrong, and I have no reason to believe the resulting distribution of errors is biased toward agreement with Western morality. If you meant to say "most of them are the ancient Greeks", then sure.

Comment author: steven 12 August 2008 01:22:10PM 0 points [-]

On second thought I suppose it could mean "being a jerk according to 'ethics'", where "ethics" is conceived not as something intrinsically moral but as a practical way for agents with different moralities to coordinate on a mutually acceptable solution.

Comment author: Caledonian2 12 August 2008 02:26:52PM 4 points [-]

The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.

Eliezer, relative to other humans, your ability to believe in a personal creative deity is damaged.

Do you want a pill to help you be religious?

Comment author: Larry_D'Anna 12 August 2008 02:45:26PM 1 point [-]

Virge: The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.

So, with respect to your specific examples:

Homosexuals: sexual preference certainly is a complex adaptation, but obviously one that differs between males and females. Homosexuals just got the wrong sexual preference for their equipment. And it doesn't do any good to say that they aren't defective. They aren't defective from a human, moral point of view, but that's not the point. From evolutions view, there's hardly anything more defective, except perhaps a fox that voluntarily restrains it's own breeding.

Stanford Prison Experiment, Psychopaths: I'm not sure if I see where the complex adaptation is here. Some people have more empathy, some less. Even if the difference is supposed to be genetic, there seem to be a lot of these flexible parameters in our genome. Empathy-level could be like skin-color, height, hairiness, etc. We all have the machinery to compute empathy (we all have the same complex adaptation), but it's used more often, or carries more influence in some people and less in others. Those that totally lack empathy are like albinos. They have the genes that are supposed to code for empathy, but they're broken.

Of course you are right that empirical data on this question is needed. But absent that, we have what looks like a strong theoretical argument for psycological unity.

Comment author: Larry_D'Anna 12 August 2008 04:06:00PM 0 points [-]

Roko: I think Eliezer has explicitly stated that he is a realist.

Comment author: Virge2 12 August 2008 04:08:00PM 2 points [-]

Larry D'Anna: "And it doesn't do any good to say that they aren't defective. They aren't defective from a human, moral point of view, but that's not the point. From evolutions view, there's hardly anything more defective, except perhaps a fox that voluntarily restrains it's own breeding."

Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.

Larry D'Anna: "I'm not sure if I see where the complex adaptation is here. Some people have more empathy, some less. Even if the difference is supposed to be genetic, there seem to be a lot of these flexible parameters in our genome."

I wasn't claiming a complex adaptation. I was claiming "other computations that could exhibit a superficial unity, but with a broad spread."

I think we are already in substantial agreement, and having seen Eliezer's last comment, I see that much of what I've been rambling on about comes from reading more than was warranted into the last paragraphs of his blog entry.

Comment author: Eliezer_Yudkowsky 12 August 2008 04:32:00PM 5 points [-]

Roko:

I would not extrapolate the volitions of people whose volitions I deem to be particularly dangerous, in fact I would probably only extrapolate the volition of a small subset (perhaps 1 thousand - 1 million) people whose outward philosophical stances on life were at least fairly similar to mine.

Then you are far too confident in your own wisdom. The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that much wiser than they.

I'm sure that Archimedes of Syracuse thought that Syracuse had lots of incredibly important philosophical and cultural differences with the Romans who were attacking his city.

Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world - due to all those incredibly important philosophical differences.

Without looking in Wikipedia, can you remember what any of those philosophical differences were?

And you are separated from Archimedes by nothing more than a handful of centuries.

Comment author: cousin_it 18 November 2010 06:49:35AM *  9 points [-]

The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that much wiser than they.

"Wiser"? What's that mean?

Your comment makes me think that, as of 12 August 2008, you hadn't yet completely given up on your dream of finding a One True Eternal Morality separate from the computation going on in our heads. Have you changed your opinion in the last two years?

Comment author: wedrifid 18 November 2010 07:18:11AM 4 points [-]

I like what Roko has to say here and find myself wary of Eliezer's reply. He may have just been signalling naivety and an irrational level of egalitarianism so people are more likely to 'let him out of the box'. Even so, taking this and the other statements EY has made on FAI behaviours (yes, those that he would unilaterally label friendly) scares me.

Comment author: cousin_it 18 November 2010 07:31:46AM *  12 points [-]

unilaterally label friendly

I love your turn of phrase, it has a Cold War ring to it.

The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes? How do we determine, in general, which things a document like CEV must spell out and which things can/should be left to the mysterious magic of "intelligence"?

Comment author: wedrifid 18 November 2010 07:53:37AM 11 points [-]

The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes?

This has been my thought exactly. Barring all but the most explicit convolution any given person would prefer their own personal volition to be extrapolated. If by happenstance I should be altruistically and perfectly infatuated by, say Sally, then that's the FAI's problem. It will turn out that extrapolating my volition will then entail extrapolating Sally's volition. The same applies to caring about 'humanity', whatever that fuzzy concept means when taken in the context of unbounded future potential.

I am also not sure how to handle those who profess an ultimate preference for a possible AI that extrapolates other than their own volition. I mean, clearly they are either lying, crazy or naive. It seems safer to trust someone who says "I would ultimately prefer FAI<someone> but I am creating FAI<larger group including wedrifid> for the purpose of effective cooperation."

Similarly, if someone wanted to credibly signal altruism to me it would be better to try to convince me that CEV<someone> has a lot of similarities with CEV<benefactor> that arise due to altruistic desires rather than saying that they truly sincerely prefer CEV<someone, benefactor>. Because the later is clearly bullshit of some sort.

How do we determine, in general, which things a document like CEV must spell out, and which things can/should be left to the mysterious magic of "intelligence"?

I have no idea, I'm afraid.

Comment author: Eugine_Nier 18 November 2010 08:29:45AM 8 points [-]

Eliezer appears to be asserting that CEV<someone> is equal for all humans. His arguments leave something to be desired. In particular, this is an assertion about human psychology, and requires evidence that is entangled with reality.

Leaving aside the question of whether even a single human's volition can be extrapolated into a unique coherent utility function, this assertion has two major components:

1) humans are sufficiently altruistic that say CEV<Alice> doesn't in any way favor Alice over Bob.

2) humans are sufficiently similar that any apparent moral disagreement between Alice and Bob is caused by one or both having false beliefs about the physical world.

I find both these statements dubious, especially the first, since I see on reason why evolution would make us that altruistic.

Comment author: Perplexed 18 November 2010 06:49:40PM 1 point [-]

Eliezer appears to be asserting that CEV<someone> is equal for all humans.

The phrase "is equal for all humans" is ambiguous. Even if all humans had identical psychologies, that could still all be selfish. The scare-quoted "source code" for Values<Eliezer> and Values<Archimedes> might be identical, but I think that both will involve self "pointers" resolving to Eliezer in one case and to Archimedes in the other.

We can define that two persons values are "parametrically identical" if they can be expressed in the same "source code", but the code contains one or more parameters which are interpreted differently for different persons. A self pointer is one obvious parameter that we might be prepared to permit in "coherent" human values. That people are somewhat selfish does not necessarily conflict with our goal of determining a fair composite CEV of mankind - there are obvious ways of combining selfish values into composite values by giving "equal weight" (more scare quotes) to the values of each person.

The question then arises, are there other parameters we should expect besides self? I believe there are. One of them can be called the now pointer - it designates the current point in time. The now pointer in Values<Archimedes> resolves to ~150 BC whereas Values<Eliezer> resolves to ~2010 AD. Both are allowed to be more interested in the present and immediate future than in the distant future. (Whether they should be interested at all in the recent past is an interesting question, but somewhat orthogonal to the present topic.)

How do we combine now pointers of different persons when constructing a CEV for mankind. Do we do it by assigning "equal weights" to the now of each person as we did for the self pointers? I believe this would be a mistake. What we really want, I believe, is a weighting scheme which changes over time - a system of exponential discounting. Actions taken by an FAI in the year 2100 should mostly be for the satisfaction of the desires of people alive in 2100. The FAI will give some consideration in 2100 to the situation in 2110 because the people around in 2100 will also be interested in 2110 to some extent. It will (in 2100) give less consideration to the prospects in 2200, because people in 2100 will be not that interested in 2200. "After all", they will rationally say to themselves, "we will be paying the year 2200 its due attention in 2180, and 2190, and especially 2199. Let the future care for itself. It certainly isn't going to care for us!"

There are various other parameters that may appear in the idealized common "source code" for Values<person>. For example, there may be different preferences regarding the discount rate used in the previous paragraph, and there may be different preferences regarding the "Malthusian factor" - how many biological descendents or clones one accumulates and how fast. It is not obvious to me whether we need to come up with rules for combining these into a CEV or whether the composite versions of these parameters fall out automatically from the rules for combining self and now parameters.

Sorry for the long response, but your comment inspired me.

Comment author: timtyler 18 November 2010 11:55:19PM *  -1 points [-]

What we really want, I believe, is a weighting scheme which changes over time - a system of exponential discounting. Actions taken by an FAI in the year 2100 should mostly be for the satisfaction of the desires of people alive in 2100. The FAI will give some consideration in 2100 to the situation in 2110 because the people around in 2100 will also be interested in 2110 to some extent. It will (in 2100) give less consideration to the prospects in 2200, because people in 2100 will be not that interested in 2200. "After all", they will rationally say to themselves, "we will be paying the year 2200 its due attention in 2180, and 2190, and especially 2199.

I don't think you need a "discounting" scheme. Or at least, you would get what is needed there "automatically" - if you just maximise expected utility. The same way Deep Blue doesn't waste its time worrying about promoting pawns on the first move of the game - even if you give it the very long term (and not remotely "discounted") goal of winning the whole game.

Comment author: Perplexed 19 November 2010 12:18:47AM 1 point [-]

I don't think you need a "discounting" scheme. Or at least, you would get what is needed there "automatically" - if you just maximise expected utility.

Could you explain why you say that? I can imagine two possible reasons why you might, but they are both wrong. Your "Deep Blue" example suggests that you are laboring under some profound misconceptions about utility theory and the nature of instrumental values.

Comment author: timtyler 19 November 2010 08:04:19AM *  -1 points [-]

This is this one again. You don't yet seem to agree with it - and it isn't clear to me why not.

Comment author: Jack 19 November 2010 12:31:36PM 0 points [-]

The same way Deep Blue doesn't waste its time worrying about promoting pawns on the first move of the game - even if you give it the very long term (and not remotely "discounted") goal of winning the whole game.

Is this really true? My understanding is that Deep Blue's position evaluation function was determined by an analysis of a hundreds of thousands of games. Presumably it ranked openings which had a tendency to produce more promotion opportunities higher than openings which tended to produce fewer promotion opportunities (all else being equal and assuming promoting pawns correlates with wins).

Comment author: timtyler 19 November 2010 08:40:45PM *  0 points [-]

I wasn't talking about that - I meant it doesn't evaluate board positions with promoted pawns at the start of the game - even though these are common positions in complete chess games. Anyway, forget that example if you don't like it, the point it illustrates is unchanged.

Comment author: timtyler 20 November 2010 10:16:04AM 1 point [-]

Eliezer appears to be asserting that CEV<someone> is equal for all humans.

The "C" in "CEV" stands for "Coherent". The concept refers to techniques of combining the wills of a bunch of agents. The idea is not normally applied to a population consisting of single human. That would just be EV<someone>. I am not aware of any evidence that Yu-El thinks that EV<someone> is independent of the <someone>.

Comment author: nshepperd 20 November 2010 11:43:14AM 2 points [-]

A CEV optimizer is less likely to do horrific things while its ability to extrapolate volition is "weak". If it can't extrapolate far from the unwise preferences people have now with the resources it has, it will notice that the EV varies a lot among the population, and take no action. Or if the extrapolation system has a bug in it, this will hopefully show up as well. So coherence is a kind of "sanity test".

That's one reason that leaps to mind anyway.

Of course the other is that there is no evidence any single human is Friendly anyway, so cooperation would be impossible among EV maximizing AI researchers. As such, an AI that maximizes EV<Eliezer> is out of the question already. CEV<humanity> is the next best thing.

Comment author: Vladimir_Nesov 18 November 2010 10:44:20AM *  3 points [-]

The argument seems to be, if Preference1<Archimedes> is too different from Preference1<cousin_it>, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2<Archimedes> much closer to Preference2<cousin_it>. And since Preference1 is inadequate, as demonstrated by this test case, Preference1<cousin_it> is also probably hugely worse for cousin_it than Preference2<Archimedes>, even if Preference2<cousin_it> is better than Preference2<Archimedes>.

We are not that wise in the sense that any moral progress we've achieved, if it's indeed progress (so that on reflection, both past and future would agree that the direction was right) and not arbitrary change, shouldn't be a problem for an AI to repeat, and thus this progress in particular (as opposed to other possible differences) shouldn't contribute to differences in extracted preference.

Comment author: Eugine_Nier 18 November 2010 04:23:37PM *  1 point [-]

The argument seems to be, if Preference1<Archimedes> is too different from Preference1<cousin_it>, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2<Archimedes> much closer to Preference2<cousin_it>. And since Preference1 is inadequate, as demonstrated by this test case, Preference1<cousin_it> is also probably hugely worse for cousin_it than Preference2<Archimedes>, even if Preference2<cousin_it> is better than Preference2<Archimedes>.

Of course the above constraint isn't nearly enough to uniquely specify Preference2.

Comment author: steven 12 August 2008 05:19:00PM 0 points [-]

Just to check, surely you're not saying an extrapolated version of Archimedes and a thousand people who agreed with him wouldn't have turned out OK?

It seems to me that we have some quite strong evidence against rotten bastards theory in that intelligent and well-informed people IRL seem to converge away from bastardly beliefs. Still, rotten bastards theory seems worth thinking about for negative-utilitarian reasons.

Comment author: Tim_Tyler 12 August 2008 05:36:00PM 1 point [-]

The basic ev-bio necessity behind the psychological unity of human brains is not widely understood.

"Necessity" may be over-stating it. Humans are not very diverse, due to recent genetic bottlenecks. We do have enormous psychological differences between us - due to neotony-induced developmental plasticity - and because there are many ways to break a human. We haven't yet lived in the human hive long enough for the developmental plasticity to result in clearly-discrete queen/worker/warrior castes, with associated mental adaptations, though.

Go back in time, and often proto-humanity was divided into multiple tribes: the neanderthals, the cro-magnons, etc - with consequently reduced psychological unity. Travel and the modern gene pool expansion and stirring has obliterated the competeting tribes.

Comment author: Tim_Tyler 12 August 2008 05:47:00PM 0 points [-]

Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world

I doubt there would have been much controversy. The fate of humanity is bound to be obliteration by far more advanced technology. The idea that our crappy evolved slug bodies, with their sluggish meat brains might go the distance would have seemed pretty ridiculous, even back then.

Comment author: Larry_D'Anna 12 August 2008 07:13:00PM 0 points [-]

Virge: Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.

well because we're trying to characterize the sort of psychological diversity that can exist within our species. And this psychological unity argument is saying "we're all the same, except for a mix of one-step changes". This means that any complex adaptation in any human is in almost all humans. The exceptions being: sexual dimorphism, and the fact that certain individuals are "defective", in the sense that one of their complex adaptations is broken. So if you're arguing against this position and saying: look at homosexuals, they are different but not broken, then you aren't talking about the same kind of "broken". I'm not arguing that we should make base any moral judgment on evolution's silly ideas of what's broken. I'm just arguing that homosexuals don't serve as a counterexample to the idea of psychological unity.

Virge: I wasn't claiming a complex adaptation. I was claiming "other computations that could exhibit a superficial unity, but with a broad spread."

I think you're right here. But hopefully the spread is not so much that we could not come to agree if we "knew more, thought faster, were more the people we wished we were, and had grown up farther together."

An example that gives me hope that this could be true is Vengeance. We seem to be born with a terminal value saying that it is good to see our enemies suffer. In terms of explicit moral argument, if not in deeds, we have mostly come to agree that it is wrong to take vengeance; even though some of us came from cultures that once reveled in it; even though some of us may be more naturally inclined towards it.

When you do Bayesian updates, the specific values of your initial priors become less important as you gather more evidence. The evidence piles up exponentially, and you need exponentially bad priors to keep the wrong answer in the face of it. Perhaps our moral debates are similar. Perhaps the "broad spread" is not so great that we cannot come to agree, if we consider enough evidence, if we hear enough arguments. Psychological unity does not prove that this is so, but without psychological unity, there would be no reason to hope it is possible.

Comment author: Doug_S. 12 August 2008 10:30:00PM 2 points [-]

Many people seem to have set their priors to 1 on several facts. I suspect even MiniLuv would have a hard time getting these guys to believe that Christianity is false.

In other words:

Bob: "Abortion is wrong."

Sally: "Do you think that this is something of which most humans ought to be persuadable?"

Bob: "Yes, I do. Do you think abortion is right?"

Sally: "Yes, I do. And I don't think that's because I'm a psychopath by common human standards. I think most humans would come to agree with me, if they knew the facts I knew, and heard the same moral arguments I've heard."

Bob and Sally are wrong. Well, half-wrong, anyway. A human's beliefs depend on the order in which he hears arguments. If I info-dumped all my experiences into the Pope's mind, and vice-versa, he'd still be a Christian and I'd still be an atheist.

I do not think that rational arguments and evidence are sufficient to convince people to change a sufficiently stubborn belief.

Eliezer, you've used your AI-box experiment as an example that people can be persuaded. If you had the Pope in a holodeck that you completely controlled, do you think you could convince him that Jesus was never resurrected? I really doubt it.

Comment author: Jadagul 13 August 2008 12:26:00AM 0 points [-]

Doug raises another good point. Related to what I said earlier, I think people really do functionally have prior probability=1 on some propositions. Or act as if they do. If "The Bible is the inerrant word of God" is a core part of your worldview, it is literally impossible for me to convince you this is false, because you use this belief to interpret any facts I present to you. Eliezer has commented before that you can rationalize just about anything; if "God exists" or "The Flying Spaghetti Monster exists" or "reincarnation exists" is part of the machinery you use to interpret your experience, in a deep enough way, your experiences can't disprove it.

Comment author: Eliezer_Yudkowsky 13 August 2008 01:12:00AM 7 points [-]

If you had the Pope in a holodeck that you completely controlled, do you think you could convince him that Jesus was never resurrected?

Oh hell yeah. Then again, I'd also go for it if you offered me 24 hours. Then again, the Pope almost certainly already believes that Jesus was never resurrected "except in our hearts" or some such, which makes the job much harder.

Comment author: Doug_S. 13 August 2008 02:19:00AM 0 points [-]

If the Pope knows that he's in a holodeck, and that you are controlling it, would that change your answer?

Comment author: Tim_Tyler 13 August 2008 07:18:00AM 2 points [-]

The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.

That argument is not right - as fig wasps demonstrate. Organisms can exhibit developmental plasticity, and show different traits under different circumstances. Humans are a good example of this: we show remarkable developmental plasticity due to neoteny. E.g. the phenotypes of the jockey and the sumo wrestler are remarkably different. The jockey is continuously on calorie restriction - and so exhibits a radically different gene expression profile in almost every tissue.

Comment author: Recovering_irrationalist 13 August 2008 01:19:00PM 0 points [-]

Eliezer: The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI.

I'd feel a lot safer if you'd extend this back at least to the infanticidal hunter-gatherers, and preferably to apes fighting around the 2001 monolith.

Comment author: Eliezer_Yudkowsky 13 August 2008 01:30:00PM 2 points [-]

Recovering, if I write CEV then it ought to work on hunter-gatherers, but the hunter-gatherers could not plausible have understood the concepts involved in CEV (whereas I might be willing to try to explain it to Archimedes, if not any other Syracusans). So "think of a strategy such that it would work for hunter-gatherers" fails here; I can't visualize that counterfactual unless the hunter-gatherers have the concept of math.

The apes fighting around the monolith are outside my moral reference frame and a CEV focused on them would produce different results.

Comment author: steven 13 August 2008 01:41:00PM 0 points [-]

Roko, it doesn't just edit incorrect factual beliefs, there's also the "what if we thought faster, were smarter, were more like the people we wished we were", etc.

Comment author: steven 13 August 2008 03:51:00PM 0 points [-]

If you think an IQ-200 extreme-polymath pope who pondered all current and future arguments for atheism would still remain a Christian, then maybe.

Comment author: Constant2 13 August 2008 05:01:00PM 2 points [-]

Time - Philip Johnson is not just a Christian but a creationist. Do you mean, "if there are smart creationists out there..."? I don't really pay much attention to the religious beliefs of the smartest mathematicians and scientists and I'm not especially keen on looking into it now, but I would be surprised if all top scientists without exception were atheists. This page seems to suggest that many of the best scientists are something other than atheist, many of those Christian.

Comment author: steven 13 August 2008 05:35:00PM 1 point [-]

PhD in mathematical logic != extreme polymath. I guess Dyson and Tipler come closest to refuting my position, but I wouldn't expect their beliefs to remain constant under a few thousand years of pondering the overcoming bias archives, say.

Comment author: Caledonian2 13 August 2008 05:58:00PM 0 points [-]

You can bring a brain to data, but you can't make it think.

Things like an IQ of 200 indicate that a person has certain cognitive strengths. They DO NOT indicate that those strengths will be utilized. Someone who is not concerned about being dishonest with themselves can self-convince of whatever they please.

At this point, there are no rational grounds for a person to accept the truth claims of, say, the various Christian doctrinal groups. A person who accepts them regardless has already decided to suspend rationality - unless a desire for honesty greater than their desire to continue to believe can be induced in them, they will never be reasoned out of their convinctions. Such induction is impossible in most people and is highly problematic in the remaining few.

Having the capacity to do a thing isn't enough - you need also the motivation to do it. Lacking that motivation, potential becomes irrelevant.

Comment author: Tim_Tyler 13 August 2008 08:06:00PM 0 points [-]

What I mean is, if there are smart Christians out there, why can't they put together a coherent argument favouring Christianity?

I mean, is this really the best they can do?

Comment author: frelkins 13 August 2008 09:17:00PM 1 point [-]

@Tim Tyler

"a coherent argument favouring Christianity"

Catholic people believe that they have such a thing, resting on strong philosophical definitions of ontology and truth. Traditionally I think that if one was truly interested in hearing a coherent argument for Christian belief, the Jesuit order specialized in teaching and expounding the philosophy. Short a long session with a Jesuit, you might consult the Catholic Encyclopedia for Christian arguments.

While perhaps you will ultimately agree that their system is coherent, that is, "marked by an orderly, logical, and aesthetically consistent relation of parts," I doubt you will cotton to their first principles.

I do agree with Collins in your Time link on one point and one point alone: sincere discussion is probably the best way for honest minds. Otherwise, he's not my demitasse of anything.

Comment author: Jadagul 14 August 2008 03:31:00AM 0 points [-]

Caledonian and Tim Tyler: there are lots of coherent defenses of Christianity. It's just that many of them rest on statements like, "if Occam's Razor comes into conflict with Revealed Truth, we must privilege the latter over the former." This isn't incoherent; it's just wrong. At least from our perspective. Which is the point I've been trying to make. They'd say the same thing about us.

Roko: I sent you an email.

Comment author: Peterdjones 09 June 2011 09:15:02PM *  0 points [-]

Well, it is not logically necessary that we have a genuine disagreement. We might be mistaken in believing ourselves to mean the same thing by the words right and wrong, since neither of us can introspectively report our own moral reference frames or unfold them fully."

I think the idea of associating the meaning of a word with a detailed theory of fine-grained set of criteria, allowing you to apply the term in all cases, has disadvantages.

Newtonian theory has a different set of fine grained criteria about gravity than relativistic theory. If we take those criteria as define the meaning of gravity, then they must be talking about different things. If we take that as the end of the story, then there is no way we can make sense of their being contrasting theories, or of one theory superseding another. The one is a theory of Newton-gravity, the other of Einstein-gravity. If we take them as both being theories of some more vaguely and coarsely defined notion gravity, we can explain their disagreement and differing success. And we don't have to give up on Einstein-gravity and Newton-gravity.

You cannot have a disagreement about which algorithm should direct your actions, without first having the same meaning of should - and no matter how you try to phrase this in terms of "what ought to direct your actions" or "right actions" or "correct heaps of pebbles", in the end you will be left with the empirical fact that it is possible to construct minds directed by any coherent utility function.

I don't see the point of this comment. No-one holds that the constraint that makes some sets of values genuinely moral is the same as the constrain that makes them implementable.

When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes.

That are both in the superclass of optimisation processes. Why should they not both be in the class of genuinely moral optimisation processes?

You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about - unless you cover "disagreement" to include differences where two agents have nothing to say to each other.

Meta ethics can supply a meaning of should/ought without specifying anything specific. For instance, if you ought to maximise happiness, that doesn't specify any action without further information about what leads to happiness.