Moral Error and Moral Disagreement

Eliezer Yudkowsky

26 Moral Error and Moral Disagreement

by Eliezer Yudkowsky

10th Aug 2008

7 min read

133

26

Followup to: Inseparably Right, Sorting Pebbles Into Correct Heaps

Richard Chappell, a pro, writes:

"When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I don't see how Eliezer can accommodate this. On his account, what Bob asserted is true iff abortion is prohibited by the morality_Bob norms. How can Sally disagree? There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob...

Since there is moral disagreement, whatever Eliezer purports to be analysing here, it is not morality."

The phenomena of moral disagreement, moral error, and moral progress, on terminal values, are the primary drivers behind my metaethics. Think of how simple Friendly AI would be if there were no moral disagreements, moral errors, or moral progress!

Richard claims, "There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob."

We may not suppose, and there is disputing. Bob does not have direct, unmediated, veridical access to the output of his own morality.

I tried to describe morality as a "computation". In retrospect, I don't think this is functioning as the Word of Power that I thought I was emitting.

Let us read, for "computation", "idealized abstract dynamic"—maybe that will be a more comfortable label to apply to morality.

Even so, I would have thought it obvious that computations may be the subjects of mystery and error. Maybe it's not as obvious outside computer science?

Disagreement has two prerequisites: the possibility of agreement and the possibility of error. For two people to agree on something, there must be something they are agreeing about, a referent held in common. And it must be possible for an "error" to take place, a conflict between "P" in the map and not-P in the territory. Where these two prerequisites are present, Sally can say to Bob: "That thing we were just both talking about—you are in error about it."

Richard's objection would seem in the first place to rule out the possibility of moral error, from which he derives the impossibility of moral agreement.

So: does my metaethics rule out moral error? Is there no disputing that abortion is indeed prohibited by morality_Bob?

This is such a strange idea that I find myself wondering what the heck Richard could be thinking. My best guess is that Richard, perhaps having not read all the posts in this sequence, is taking my notion of morality_Bob to refer to a flat, static list of valuations explicitly asserted by Bob. "Abortion is wrong" would be on Bob's list, and there would be no disputing that.

But on the contrary, I conceive of morality_Bob as something that unfolds into Bob's morality—like the way one can describe in 6 states and 2 symbols a Turing machine that will write 4.640 × 10¹⁴³⁹ 1s to its tape before halting.

So morality_Bob refers to a compact folded specification, and not a flat list of outputs. But still, how could Bob be wrong about the output of his own morality?

In manifold obvious and non-obvious ways:

Bob could be empirically mistaken about the state of fetuses, perhaps believing fetuses to be aware of the outside world. (Correcting this might change Bob's instrumental values but not terminal values.)

Bob could have formed his beliefs about what constituted "personhood" in the presence of confusion about the nature of consciousness, so that if Bob were fully informed about consciousness, Bob would not have been tempted to talk about "the beginning of life" or "the human kind" in order to define personhood. (This changes Bob's expressed terminal values; afterward he will state different general rules about what sort of physical things are ends in themselves.)

So those are the obvious moral errors—instrumental errors driven by empirical mistakes; and erroneous generalizations about terminal values, driven by failure to consider moral arguments that are valid but hard to find in the search space.

Then there are less obvious sources of moral error: Bob could have a list of mind-influencing considerations that he considers morally valid, and a list of other mind-influencing considerations that Bob considers morally invalid. Maybe Bob was raised a Christian and now considers that cultural influence to be invalid. But, unknown to Bob, when he weighs up his values for and against abortion, the influence of his Christian upbringing comes in and distorts his summing of value-weights. So Bob believes that the output of his current validated moral beliefs is to prohibit abortion, but actually this is a leftover of his childhood and not the output of those beliefs at all.

(Note that Robin Hanson and I seem to disagree, in a case like this, as to exactly what degree we should take Bob's word about what his morals are.)

Or Bob could believe that the word of God determines moral truth and that God has prohibited abortion in the Bible. Then Bob is making metaethical mistakes, causing his mind to malfunction in a highly general way, and add moral generalizations to his belief pool, which he would not do if veridical knowledge of the universe destroyed his current and incoherent metaethics.

Now let us turn to the disagreement between Sally and Bob.

You could suggest that Sally is saying to Bob, "Abortion is allowed by morality_Bob", but that seems a bit oversimplified; it is not psychologically or morally realistic.

If Sally and Bob were unrealistically sophisticated, they might describe their dispute as follows:

Bob: "Abortion is wrong."

Sally: "Do you think that this is something of which most humans ought to be persuadable?"

Bob: "Yes, I do. Do you think abortion is right?"

Sally: "Yes, I do. And I don't think that's because I'm a psychopath by common human standards. I think most humans would come to agree with me, if they knew the facts I knew, and heard the same moral arguments I've heard."

Bob: "I think, then, that we must have a moral disagreement: since we both believe ourselves to be a shared moral frame of reference on this issue, and yet our moral intuitions say different things to us."

Sally: "Well, it is not logically necessary that we have a genuine disagreement. We might be mistaken in believing ourselves to mean the same thing by the words right and wrong, since neither of us can introspectively report our own moral reference frames or unfold them fully."

Bob: "But if the meaning is similar up to the third decimal place, or sufficiently similar in some respects that it ought to be delivering similar answers on this particular issue, then, even if our moralities are not in-principle identical, I would not hesitate to invoke the intuitions for transpersonal morality."

Sally: "I agree. Until proven otherwise, I am inclined to talk about this question as if it is the same question unto us."

Bob: "So I say 'Abortion is wrong' without further qualification or specialization on what wrong means unto me."

Sally: "And I think that abortion is right. We have a disagreement, then, and at least one of us must be mistaken."

Bob: "Unless we're actually choosing differently because of in-principle unresolvable differences in our moral frame of reference, as if one of us were a paperclip maximizer. In that case, we would be mutually mistaken in our belief that when we talk about doing what is right, we mean the same thing by right. We would agree that we have a disagreement, but we would both be wrong."

Now, this is not exactly what most people are explicitly thinking when they engage in a moral dispute—but it is how I would cash out and naturalize their intuitions about transpersonal morality.

Richard also says, "Since there is moral disagreement..." This seems like a prime case of what I call naive philosophical realism—the belief that philosophical intuitions are direct unmediated veridical passports to philosophical truth.

It so happens that I agree that there is such a thing as moral disagreement. Tomorrow I will endeavor to justify, in fuller detail, how this statement can possibly make sense in a reductionistic natural universe. So I am not disputing this particular proposition. But I note, in passing, that Richard cannot justifiably assert the existence of moral disagreement as an irrefutable premise for discussion, though he could consider it as an apparent datum. You cannot take as irrefutable premises, things that you have not explained exactly; for then what is it that is certain to be true?

I cannot help but note the resemblance to Richard's assumption that "there's no disputing" that abortion is indeed prohibited by morality_Bob—the assumption that Bob has direct veridical unmediated access to the final unfolded output of his own morality.

Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob, and allowed by morality_Sally, there being at least two possible minds for whom this would be true. Then the two minds might be mistaken about believing themselves to disagree. Actually they would simply be directed by different algorithms.

You cannot have a disagreement about which algorithm should direct your actions, without first having the same meaning of should—and no matter how you try to phrase this in terms of "what ought to direct your actions" or "right actions" or "correct heaps of pebbles", in the end you will be left with the empirical fact that it is possible to construct minds directed by any coherent utility function.

When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about—unless you cover "disagreement" to include differences where two agents have nothing to say to each other.

But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it. Now, perhaps some psychopaths would not be persuadable in-principle to take the pill that would, by our standards, "fix" them. But I note the possibility to emphasize what an extreme statement it is to say of someone:

"We have nothing to argue about, we are only different optimization processes."

That should be reserved for paperclip maximizers, not used against humans whose arguments you don't like.

Part of The Metaethics Sequence

Next post: "Abstracted Idealized Dynamics"

Previous post: "Sorting Pebbles Into Correct Heaps"

Metaethics

Personal Blog

26

Mentioned in

85[Valence series] 2. Valence & Normativity

37Abstracted Idealized Dynamics

37Claiming Connotations

6[SEQ RERUN] Moral Error and Moral Disagreement

Moral Error and Moral Disagreement

0Recovering_irrationalist

New Comment

133 comments, sorted by

oldest

Click to highlight new comments since: Today at 8:16 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Nick_Tarleton16y50

Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.

3benelliott13y

I believe that zero out of 6.5 billion humans are unicorns. The big number does not create an implausibility all on its own, there may be stable psychopaths, but I wouldn't be very surprised if there weren't.

6Arandur13y

Far less likely to suppose that 1 out of 6.5 billion humans is a stable psychopath.

[-]steven16y30

I wonder if the distinction between 1) something implementing the same dynamic as a typical human but mistaken about what it says and 2) something implementing a completely different dynamic and not mistaken about anything, is the same as the distinction people normally make between 1) immoral and 2) amoral.

[-]Jadagul16y10

Steven: quite possibly related. I don't think they're exactly the same (the classic comic book/high fantasy "I'm evil and I know it" villain fits A2, but I'd describe him as amoral), but it's an interesting parallel.

Eliezer: I'm coming more and more to the conclusion that our main area of disagreement is our willingness to believe that someone who disagrees with us really "embodies a different optimization process." There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processe... (read more)

[-]TGGP416y00

If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all. A General AI should be able to convince it just about anything, right?

[-]Eliezer Yudkowsky16y60

Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.

Heck, what about babies? What do they want, and would they be complicated enough to want anything different if they knew more and thought faster?

There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processes; while I believe mine to be the best I've found, I remain aware that if I held any of the others I would believe exactly the same thing.

You would not believe exactly the same thing. If you ... (read more)

[-]conchis16y00

I must be starting to get it. That unpacked in exactly the way I expected.

On the other hand, this:

If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all.

makes no sense to me at all.

[-]Sebastian_Hagen216y00

I think my highest goal in life is to make myself happy. Because I'm not a sociopath making myself happy tends to involve having friends and making them happy. But the ultimate goal is me.

If you had a chance to take a pill which would cause you to stop caring about your friends by permanently maxing out that part of your hapiness function regardless of whether you had any friends, would you take it?
Do non-psychopaths that given the chance would self-modify into psychopaths fall into the same moral reference frame as stable psychopaths?

[-]jsalvatier16y10

Sebastian Hagen:

My intuition is that a good deal of people would take the psychopath pill. At least if the social consequences were minimal, which is besides the point.

[-]Hopefully_Anonymous16y20

Frame it defensively rather than offensively and a whole heck of a lot of people would take that pill. Of course some of us would also take the pill that negates the effects of our friends taking the first pill, hehehe.

[-]Hopefully_Anonymous16y00

Weird, jsalvati is not my sock puppet, but the 11:16pm post above is mine.

[-]Eliezer Yudkowsky16y00

Fixed. How very odd. (And for the record, jsalvati's IP is not HA's.)

[-]Richard416y20

"Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob..."

That's right. (I didn't mean to suggest that there's never any disputing what someone's moral commitments are; just that this wasn't supposed to be in dispute in the particular case I was imagining.) I take it that Sally and Bob could disagree even so, and not merely be talking past each other, even if one or both of them was impervious to rational argument. It is at least a significant cost of your theory that it denies this datum. (It doesn't have... (read more)

[-]Mike_Blume16y20

I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you.

Do you imagine that there are facts about the brains of cattle which we could both learn - facts drawn from fMRI scans, or from behavioral science experiments, perhaps - which would bring us into agreement on the issue?

3Unknowns15y

No. I agree with those facts and think that eating steak is a good idea anyway.

0Eliezer Yudkowsky15y

I'm curious: Would you eat Yoda? BTW, if you're the original Unknown, please see Open Thread for a query about our outstanding bet.

0Unknowns15y

I would eat sentient creatures but not intelligent creatures. And even in the case of sentient non-intelligent ones it depends on what I'm used to.

2Unknowns15y

By the way, is there any way to get back my original username? I didn't know how to do it when things were moved over from OB.

[-]Jadagul16y00

Eliezer: for 'better' vs 'frooter,' of course you're right. I just would have phrased it differently; I've been known to claim that the word 'better' is completely meaningless unless you (are able to) follow it with "better at or for something." So of course, Jadagul_real would say that his worldview is better for fulfilling his values. And Jadagul_hypothetical would say that his worldview is better for achieving his values. And both would (potentially) be correct. (or potentially wrong. I never claimed to be infallible, either in reality o... (read more)

[-]Tim_Tyler16y00

Re: "We have nothing to argue about, we are only different optimization processes." That seems to apply in the case when a man wants to rescue copies of his genes from eternal oblivion - by convincing his mate not to abort his prospective offspring. Of course, not many would actually say that under those circumstances.

[-]Virge216y20

Eliezer: "But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it."

How sure are you that most human moral disagreements are attributable to

lack of veridical information, or
lack of

... (read more)

[-]John_T._Kennedy16y00

Mike,

"I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you."

Accepting your propositions for the sake of argument, I still find that eating steak seems reasonable.

[-]Carl_Shulman16y20

"Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents."

Richard,

If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'

[-]Tim_Tyler16y00

Other moral issues where there are a gender differences include: "should prostitution be legalised" and "should there be tighter regulation of pornography".

Again, it seems that part of the effect is due to people's idea of what is right being influenced by their own personal role - i.e. the "different optimization processes" effect.

Gender is the most obvious source of such issues, but I'm sure you can find them in other areas of life. Race politics, for instance.

[-]TGGP416y10

People in general do not want to be shot. The person doing the shooting, the lethal weapon being fired, the location in which the shooting occurs and the time of day are all pretty much irrelevant. You can ask people if they want to be shot and they'll say no, without even specifying those details. That seems a very different case from Bob, who is considering a moral proposition and outright rejecting it.

[-]Simon_M16y00

Given all of Bacon's idols of the Mind can you ever know definitely if there is an error in your own reasoning, let alone the other persons?

You cannot rely on your moral intuition, nor the cultural norms of your time, nor academic authority, nor your internal reasoning or ability to know the soundness of your argument.

Socialization, severe biases, faulty reasoning can all make you think you are ‘correct’, but can leave you with the incorrect impression of the ‘correctness’ of your thinking. & even if presented with all the correct or relevant information some people still make these errors, so if they can so could you.

[-]Roko16y10

Eliezer: "When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about - unless you cover "disagreement" to include differences where two agents have nothing to say to each other.

But this would be an extreme position to take with respect to your fellow... (read more)

[-]Roko16y00

I should qualify this statement: "As such, we shouldn't expect much more moral agreement from humans than from rational (or approximately rational) AIs."

to instead read:

"As such, on ethical questions that had no precedent in our EEA, we shouldn't expect much more moral agreement from humans than from rational (or approximately rational) AIs, apart, of course, from the fact that most humans share a common set of cognitive biases"

one can see that this is true by looking at the vast disagreements between different moral philosophers consequentialists vs. deontological ethicists, or atheists vs. christians vs. muslims, or libertarians vs. liberals vs. communists.

[-]Will_Pearson16y00

You have been priming people to think in terms of functions. (Pure) Functions do not change. They map an input to an output, and can be substituted by a list, e.g. a function that tests for primality can can be performed by an (infinitely) long list.

You may want to describe impure functions for people without a functional programming background if you want to keep terminology like morality_john().

[-]Schizo16y00

Virge's point seems particularly important, and hopefully Eliezer can address it.

[-]Zubon16y00

I find Roko on-point. The psychological unity of humankind is important, but it can be over-stated. While human beings may occupy a very small area in the space of all possible minds, it is still an area and not a single point. When we shut up and multiply by sufficiently large numbers, very small differences in the starting point are very meaningful. If we are talking about a difference valued at 0.0000001% of a human life, and you extrapolate it over a billion lives, we are talking about life and death matters. Successful AI will affect more than a ... (read more)

[-]Eliezer Yudkowsky16y50

Virge:

How sure are you that most human moral disagreements are attributable to - lack of veridical information, or - lack of ability/tools to work through that information, or - defects? You talk freely about psychopaths and non-psychopaths as though these were distinct categories of non-defective and defective humans. I know you know this is not so. The arguments about psychological unity of humankind only extend so far. e.g., would you be prepared to tell a homosexual that, if they were fully informed, they would decide to take a pill to change their or

... (read more)

[-]steven16y00

I must be missing something -- why would you advocate something that you know you can't justify to anyone else?

[-]Zubon16y70

I said "damaged" not "missing". The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.

Does that work in the other direction? The notion is that you are your current self, but one day I inform you that, r... (read more)

[-]Sebastian_Hagen216y00

Jadagul:

But my moral code does include such statements as "you have no fundamental obligation to help other people." I help people because I like to.

While I consider myself an altruist in principle (I have serious akrasia problems in practice), I do agree with this statement. Altruists don't have any obligation to help people, it just often makes sense for them to do so; sometimes it doesn't, and then the proper thing for them is not to do it.

Roko:

In the modern world, people have to make moral choices using their general intelligence, because th

... (read more)

[-]Sebastian_Hagen216y00

Quick correction: s/abstract rational reasoning/abstract moral reasoning/

[-]Larry_D'Anna16y00

Will Pearson: Why not just treat them as pure functions in the State monad?

[-]Tim_Tyler16y10

Re: If there are distinct categories of human transpersonal values, I would expect them to look like "male and female babies", "male children", "male adults", "female children", "female adults", "neurological damage 1", "neurological damage 2", not "Muslims vs. Christians!"

That seems like the position you would get if you thought that cultural evolution could not affect people's values.

[-]Richard416y-10

Carl - "If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'"

I don't mean to def... (read more)

[-]Z._M._Davis16y10

"If there are distinct categories of human transpersonal values, I would expect them to look like [...] 'male adults', [...] 'female adults', [...] not 'Muslims vs. Christians!'"

Really? In the ways that are truly important, don't you think you have more in common with Natasha Vita-More than Osama bin Laden?

[-]steven16y00

ZM, he means after volition-extrapolating.

[-]Z._M._Davis16y20

Steven, even so, I think the basic question stands. Why should cultural differences and within-sex individual differences wash out of the CEV?

[-]steven16y00

Supposedly genetics allows for people of different ages or sexes to have different mental machinery, whereas individual genetic differences just represent low-complexity differences in tweaking. I'm not sure why that makes Eliezer's point though, if the aforementioned differences in tweaking mean different complex machinery gets activated. Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in.

[-]Roko16y00

Zubon: "if you do not think you should be a bit more sociopathic, what are the odds you have exactly the right amounts of empathy and altruism?"

that was exactly what I was going to say ; - ) My personal "gut" feeling is that have exactly the right amount of empathy, no more, no less. Intellectually, I realize that this is means I have failed the reversal test, and that I am therefore probably suffering from status quo bias. (assuming, as I believe, that there is an objectively true ethics. If you're a moral relativist, you get a blank

... (read more)

[-]Z._M._Davis16y30

Steven: "Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in."

I suspect a category error here hinging around personal identity. We say "if I had grown up in a different culture ..." when I think we mean "if the baby that grew into me had grown up in a different culture..." If the baby that grew into me had grown up in a radically different culture, I don't think ve'd be me in any meaningful sense, although of course there would be many similarities ... (read more)

[-]Tim_Tyler16y00

Re: Roko's "more reasonable position":

Human psychological unity, to the extent that it exists, includes brain speed, brain degree-of-parallelism, brain storage capacity, brain reliability - and other limitations that have little to do with human goals.

[-]Bernard_Guerrero216y00

I'm not sure this proves anything. I'll take Jadagul's description as my own. I'm maximizing my happiness, but my non-sociopath status means that I like having friends/loved ones, so maximizing my own happiness entails caring about them to some degree or another. Under my existing moral code I wouldn't take the pill, but that's because it will... (read more)

[-]Doug_S.16y30

I, too, wonder if the "psychological unity of humankind" has been a bit overstated. All [insert brand and model here] computers have identical hardware, but you can install different software on them. We're running different

Consider the case of a "something maximizer". It's given an object, and then maximizes the number of copies of that object.

You give one something maximizer a paperclip, and it becomes a paperclip maximizer. You give another a pencil, and that one becomes a pencil maximizer.

There's no particular reason to expect the p... (read more)

[-]Virge216y50

Eliezer: "The basic ev-bio necessity behind the psychological unity of human brains is not widely understood."

I agree. And I think you've over-emphasized the unity and ignored evidence of diversity, explaining it away as defects.

Eliezer: "And even more importantly, the portion of our values that we regard as transpersonal, the portion we would intervene to enforce against others, is not all of our values; it's not going to include a taste for pepperoni pizza, or in my case, it's not going to include a notion of heterosexuality or homosexuali... (read more)

[-]roko316y00

virge makes a very good point here. The human mind is probably rather flexible in terms of it's ethical views; I suspect that Eli is overplaying our psychological unity.

[-]Eliezer Yudkowsky16y50

I think that views are being attributed to me that I do not possess - perhaps on the Gricean notion that if someone attacks me for holding these views, I ought to hold them; but on a blog like this one, that leaves you prey to everyone else's misunderstanding.

I do not assert that all humans end up in the same moral frame of reference (with regard to any particular extrapolation method). I do think that psychological unity is typically underestimated, and I have a hard time taking modern culture at face value (we're the ancient Greeks, guys, not a finished... (read more)

[-]steven16y00

"Being a jerk" here means "being a jerk according to other people's notion of morality, but not according to my own notion of morality", right?

I sort of take offense to "we're the ancient Greeks"; I make sure to disagree with Western morality whenever it's wrong, and I have no reason to believe the resulting distribution of errors is biased toward agreement with Western morality. If you meant to say "most of them are the ancient Greeks", then sure.

[-]steven16y00

On second thought I suppose it could mean "being a jerk according to 'ethics'", where "ethics" is conceived not as something intrinsically moral but as a practical way for agents with different moralities to coordinate on a mutually acceptable solution.

[-]Caledonian216y50

The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.

Eliezer, relative to other humans, your ability to believe in a personal creative deity is damaged.

Do you want a pill to help you be religious?

[-]Larry_D'Anna16y10

Virge: The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.

So, with respect to your specific examples:

Homosexuals: sexual preference certainly is a complex adaptation, but obvio... (read more)

[-]Roko16y00

Eli: "I do not assert that all humans end up in the same moral frame of reference (with regard to any particular extrapolation method). I do think that psychological unity is typically underestimated,"

- right, thanks for the clarification.

"But if you read "Coherent Extrapolated Volition" you'll see that it's specifically designed to handle, among other problems, the problem of, "What if we don't all want the same thing?" What then can an AI programmer do that does not constitute being a jerk? That was my attempt to answ... (read more)

[-]Larry_D'Anna16y00

Roko: I think Eliezer has explicitly stated that he is a realist.

[-]Virge216y20

Larry D'Anna: "And it doesn't do any good to say that they aren't defective. They aren't defective from a human, moral point of view, but that's not the point. From evolutions view, there's hardly anything more defective, except perhaps a fox that voluntarily restrains it's own breeding."

Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.

Larr... (read more)

[-]Eliezer Yudkowsky16y50

Roko:

I would not extrapolate the volitions of people whose volitions I deem to be particularly dangerous, in fact I would probably only extrapolate the volition of a small subset (perhaps 1 thousand - 1 million) people whose outward philosophical stances on life were at least fairly similar to mine.

Then you are far too confident in your own wisdom. The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that mu... (read more)

[-]cousin_it14y100

The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that much wiser than they.

"Wiser"? What's that mean?

Your comment makes me think that, as of 12 August 2008, you hadn't yet completely given up on your dream of finding a One True Eternal Morality separate from the computation going on in our heads. Have you changed your opinion in the last two years?

4wedrifid14y

I like what Roko has to say here and find myself wary of Eliezer's reply. He may have just been signalling naivety and an irrational level of egalitarianism so people are more likely to 'let him out of the box'. Even so, taking this and the other statements EY has made on FAI behaviours (yes, those that he would unilaterally label friendly) scares me.

[-]cousin_it14y140

unilaterally label friendly

I love your turn of phrase, it has a Cold War ring to it.

The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes? How do we determine, in general, which things a document like CEV must spell out and which things can/should be left to the mysterious magic of "intelligence"?

[-]wedrifid14y120

The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes?

This has been my thought exactly. Barring all but the most explicit convolution any given person would prefer their own personal volition to be extrapolated. If by happenstance I should be altruistically and perfectly infatuated by, say Sally, then that's the FAI's problem. It will turn out that extrapolating my volition will then entail extrapolating Sally's volition. The same applies to caring about 'humanity', whatever that fuzzy concept means when taken in the context of unbounded future potential.

I am also not sure how to handle those who profess an ultimate preference for a possible AI that extrapolates other than their own volition. I mean, clearly they are either lying, crazy or naive. It seems safer to trust someone who says "I would ultimately prefer FAI but I am creatin... (read more)

9Eugine_Nier14y

Eliezer appears to be asserting that CEV is equal for all humans. His arguments leave something to be desired. In particular, this is an assertion about human psychology, and requires evidence that is entangled with reality. Leaving aside the question of whether even a single human's volition can be extrapolated into a unique coherent utility function, this assertion has two major components: 1) humans are sufficiently altruistic that say CEV doesn't in any way favor Alice over Bob. 2) humans are sufficiently similar that any apparent moral disagreement between Alice and Bob is caused by one or both having false beliefs about the physical world. I find both these statements dubious, especially the first, since I see on reason why evolution would make us that altruistic.

0Perplexed14y

The phrase "is equal for all humans" is ambiguous. Even if all humans had identical psychologies, that could still all be selfish. The scare-quoted "source code" for Values and Values might be identical, but I think that both will involve self "pointers" resolving to Eliezer in one case and to Archimedes in the other. We can define that two persons values are "parametrically identical" if they can be expressed in the same "source code", but the code contains one or more parameters which are interpreted differently for different persons. A self pointer is one obvious parameter that we might be prepared to permit in "coherent" human values. That people are somewhat selfish does not necessarily conflict with our goal of determining a fair composite CEV of mankind - there are obvious ways of combining selfish values into composite values by giving "equal weight" (more scare quotes) to the values of each person. The question then arises, are there other parameters we should expect besides self? I believe there are. One of them can be called the now pointer - it designates the current point in time. The now pointer in Values resolves to ~150 BC whereas Values resolves to ~2010 AD. Both are allowed to be more interested in the present and immediate future than in the distant future. (Whether they should be interested at all in the recent past is an interesting question, but somewhat orthogonal to the present topic.) How do we combine now pointers of different persons when constructing a CEV for mankind. Do we do it by assigning "equal weights" to the now of each person as we did for the self pointers? I believe this would be a mistake. What we really want, I believe, is a weighting scheme which changes over time - a system of exponential discounting. Actions taken by an FAI in the year 2100 should mostly be for the satisfaction of the desires of people alive in 2100. The FAI will give some consideration in 2100 to the situation in 2110 because the people around in 2100

0timtyler14y

I don't think you need a "discounting" scheme. Or at least, you would get what is needed there "automatically" - if you just maximise expected utility. The same way Deep Blue doesn't waste its time worrying about promoting pawns on the first move of the game - even if you give it the very long term (and not remotely "discounted") goal of winning the whole game.

0Perplexed14y

Could you explain why you say that? I can imagine two possible reasons why you might, but they are both wrong. Your "Deep Blue" example suggests that you are laboring under some profound misconceptions about utility theory and the nature of instrumental values.

-2timtyler14y

This is this one again. You don't yet seem to agree with it - and it isn't clear to me why not.

0Perplexed14y

Nor is it clear to me why you did not respond to my question / request for clarification.

1timtyler14y

I did respond. I didn't have an essay on the topic prepared - but Yu-El did, so I linked to that. If you want to hear it in my own words: Wiring in temporal discounting is usually bad - since the machine can usually figure out what temporal discounting is appropriate for its current circumstances and abilities much better than you can. It is the same as with any other type of proximate goal. Instead you are usually best off just telling the machine your preferences about the possible states of the universe. If you are thinking you want the machine to mirror your own preferences, then I recommend that you consider carefully whether your ultimate preferences include temporal discounting - or whether all that is just instrumental.

1Perplexed14y

I don't see how. My question was: Referring to this that you said: You have still not explained why you said this. The question that discounting answers is, "Which is better: saving 3 lives today or saving 4 lives in 50 years?" Which is the same question as "Which of the two has the higher expected utility in current utilons?" We want to maximize expected current utility regardless of what we decide regarding discounting. However, since you do bring up the idea of maximizing expected utility, I am very curious how you can simultaneously claim (elsewhere on this thread) that utilities are figures of merit attached to actions rather than outcomes. Are you suggesting that we should be assessing our probability distribution over actions and then adding together the products of those probabilities with the utility of each action?

2timtyler14y

Many factors "automatically" lead to temporal discounting if you don't wire it in. The list includes: * Agents are mortal - they might die before the future utility arrives * Agents exhibit senescence - the present is more valuable to them than the future, because they are younger and more vital; * The future is uncertain - agents have limited capacities to predict the future; * The future is hard to predicably influence by actions taken now; I think considerations such as the ones listed above adequately account for most temporal discounting in biology - though it is true that some of it may be the result of adaptations to deal with resource-limited cognition, or just plain stupidity. Note that the list is dominated by items that are a function of the capabilities and limitations of the agent in question. If the agent conquers senescence, becomes immortal, or improves its ability to predict or predictably influence the future, then the factors all change around. This naturally results in a different temporal discounting scheme - so long as it has not previously been wired into the agent by myopic forces. Basically, temporal discounting can often usefully be regarded as instrumental. Like energy, or gold, or warmth. You could specify how much each of these things is valued as well - but if you don't they will be assigned instrumental value anyway. Unless you think you know their practical value better than a future superintelligent agent, perhaps you are better off leaving such issues to it. Tell the agent what state of affairs you actually want - and let it figure out the details of how best to get it for you. Temporal discounting contrasts with risk aversion in this respect.

0Perplexed14y

Quite true. I'm glad you included that word "often". Now we can discuss the real issue: whether that word "often" should be changed to "always" as EY and yourself seem to claim. Or whether utility functions can and should incorporate the discounting of the value of temporally distant outcomes and pleasure-flows for reasons over and above considerations of instrumentality. A useful contrast/analogy. You seem to be claiming that risk aversion is not purely instrumental; that it can be fundamental; that we need to ask agents about their preferences among risky alternatives, rather than simply axiomatizing that a rational agent will be risk neutral. But I disagree that this is in contrast to the situation with temporal discounting. We need to allow that rational and moral agents may discount the value of future outcomes and flows for fundamental, non-instrumental reasons. We need to ask them. This is particularly the case when we consider questions like the moral value of a human life. The question before us is whether I should place the same moral value now on a human life next year and a human life 101 years from now. I say 'no'; EY (and you?) say yes. What is EY's justification for his position? Well, he might invent a moral principle that he might call "time invariance of moral value" and assert that this principle absolutely forces me to accept the equality: * value@t(life@t+1) = value@t(life@t+101). I would counter that EY is using the invalid "strong principle of time invariance". If one uses the valid "weak principle of time invariance" then all that we can prove is that: * value@t(life@t+1) = value@t+100(life@t+101) So, we need another moral principle to get to where EY wants to go. EY postulates that the moral discount rate must be zero. I simply reject this postulate (as would the bulk of mankind, if asked). EY and I can both agree to a weaker postulate, "time invariance of moral preference". But this only shows that the discounting must be exponenti

2timtyler14y

You can specify a method temporal discounting if you really want to. Just as you can specify a value for collecting gold atoms if you really want to. However, there are side effects and problems associated with introducing unnecessary constraints. If we think that such creatures are common and if we are trying to faithfully mirror and perpetuate their limitations, you mean. I don't really see this as a "should" question. However, there are consequences to wiring in instrumental values. You typically wind up with a handicapped superintelligence. I thought I already gave this as my reasoning, with comments such as "unless you think you know their practical value better than a future superintelligent agent, perhaps you are better off leaving such issues to it." Not a practical issue - IMO. We are resource-limited creatures, who can barely see 10 years into the future. Instrumental temporal discounting protects us from infinite maths with great effectiveness. This is the same as in biology. Organisms act as though they want to become ancestors - not just parents or grandparents. That is the optimisation target, anyway. However, instrumental temporal discounting protects them from far-future considerations with great effectiveness.

1Perplexed14y

You did indeed. I noticed it, and meant to clarify that I am not advocating any kind of "wiring in". Unfortunately, I failed to do so. My position would be that human beings often have discount factors "wired in" by evolution. It is true, of course, that like every other moral instinct analyzed by EvoPsych, the ultimate adaptationist evolutionary explanation of this moral instinct is somewhat instrumental, but this doesn't make it any less fundamental from the standpoint of the person born with this instinct. As for moral values that we insert into AIs, these too are instrumental in terms of their final cause - we want the AIs to have particular values for our own instrumental reasons. But, for the AI, they are fundamental. But not necessarily 'wired in'. If we, as I believe we should, give the AI a fundamental meta-value that it should construct its own fundamental values by empirically constructing some kind of CEV of mankind - if we do this then the AI will end up with a discount factor because his human models have discount factors. But it won't be a wired-in or constant discount factor. Because the discount factors of mankind may well change over time as the expected lifespan of humans changes, as people upload and choose to run at various rates, as people are born or as they die. I'm saying that we need to allow for an AI discount factor or factors which are not strictly instrumental, but which are not 'wired in' either. And especially not a wired-in discount factor of exactly zero!

0timtyler14y

I think we want a minimally myopic superintelligence - and fairly quickly. We should not aspire to program human limitations into machines - in a foolish attempt to mirror their values. If the Met. Office computer is handling orders asking it to look three months out - and an ethtics graduate says that it too future-oriented for a typical human, and it should me made to look less far out, so it better reflects human values - he should be told what an idiot he is being. We use machines to complement human capabilities, not just to copy them. When it comes to discounting the future, machines will be able to see and influence furtther - and we would be well-advised let them. Much harm is done today due to temporal discounting. Governments look no further than election day. Machines can help put a stop to such stupidity and negligence - but we have to know enough to let them. As Eleizer says, he doesn't propose doing much temporal discounting - except instrumentally. That kind of thing can be expected to go up against the wall as part of the "smarter, faster, wiser, better" part of his CEV.

-1Perplexed14y

And so we are in disagreement. But I hope you now understand that the disagreement is because our values are different rather than because I don't understand the concept of values. Ironically our values differ in that I prefer to preserve my values and those of my conspecifics beyond the Singularity, whereas you distrust those values and the flawed cognition behind them, and you wish to have those imperfect human things replaced by something less messy.

0timtyler14y

I don't see myself as doing any non-instrumental temporal discounting in the first place. So, for me personally, losing my non-instrumental temporal discounting doesn't seem like much of a loss. However, I do think that our temporal myopia is going to fall by the wayside. We will stop screwing over the immediate future because we don't care about it enough. Myopic temporal discounting represents a primitive form of value - which is destined to go the way of cannibalism and slavery.

2timtyler14y

Regarding utility, utilities are just measures of satisfaction. They can be associated with anything. It is a matter of fact that utilities are associated with actions in most agents - since agents have evolved to calculate utilities in order to allow them to choose between their possible actions. I am not claiming that utilities are not frequently associated with outcomes. Utilities are frequently linked to outcomes - since most evolved agents are made so in such a way that they like to derive satisfaction by manipulating the external world. However, nowhere in the definition of utility does it say that utilities are necessarily associated with external-world outcomes. Indeed, in the well-known phenomena of "wireheading" and "drug-taking" utility is divorced from external-world outcomes - and deliberately manufactured.

0Perplexed14y

True. But in most economic analysis, terminal utilities are associated with outcomes; the expected utilities that become associated with actions are usually instrumental utilities. Nevertheless, I continue to agree with you that in some circumstances, it makes sense to attach terminal utilities to actions. This shows up, for example, in discussions of morality from a deontological viewpoint. For example, suppose you have a choice of lying or telling the truth. You assess the consequences of your actions, and are amused to discover that there is no difference in the consequences - you will not be believed in any case. A utilitarian would say that there is no moral difference in this case between lying and telling the truth. A Kant disciple would disagree. And the way he would explain this disagreement to the utilitarian would be to attach a negative moral utility to the action of speaking untruthfully.

2timtyler14y

Utilities are often associated with states of the world, yes. However, here you seemed to balk at utilities that were not so associated. I think such values can still be called "utilities" - and "utility functions" can be used to describe how they are generated - and the standard economic framework accommodates this just fine. What this idea doesn't fit into is the von Neumann–Morgenstern system - since it typically violates the independence axiom. However, that is not the end of the world. That axiom can simply be binned - and fairly often it is.

0Perplexed14y

Unless you supply some restrictions, it is considerably more destructive than that. All axioms based on consequentialism are blown away. You said yourself that we can assign utilities so as to rationalize any set of actions that an agent might choose. I.e. there are no irrational actions. I.e. decision theory and utility theory are roughly as useful as theology.

-2timtyler14y

No, no! That is like saying that a universal computer is useless to scientists - because it can be made to predict anything! Universal action is a useful and interesting concept partly because it allows a compact, utility-based description of arbitrary computable agents. Once you have a utility function for an agent, you can then combine and compare its utility function with that of other agents, and generally use the existing toolbox of economics to help model and analyse the agent's behaviour. This is all surely a Good Thing.

0Perplexed14y

I've never seen the phrase universal action before. Googling didn't help me. It certainly sounds like it might be an interesting concept. Can you provide a link to an explanation more coherent than the one you have attempted to give here? As to whether a "utility-based" description of an agent that does not adhere to the standard axioms of utility is a "good thing" - well I am doubtful. Surely it does not enable use of the standard toolbox of economics, because that toolbox takes for granted that the participants in the economy are (approximately) rational agents.

0timtyler14y

Universal action is named after universal computation and universal construction. Universal construction and universal action have some caveats about being compatible with constraints imposted by things like physical law. "Doing anything" means something like: being able to feed arbitrary computable sequences in parallel to your motor outputs. Sequences that fail due to severing your own head don't violate the spirit of the idea, though. As with universal computation, universal action is subject to resource limitations in practice. My coinage - AFAIK. Attribution: unpublished manuscript ;-)

0Perplexed14y

Well, I'll just ignore the fact that universal construction means to me something very different than it apparently means to you. Your claim seems to be that we can 'program' a machine (which is already known to maximize utility) so as to output any sequence of symbols we wish it to output; program it by the clever technique of assigning a numeric utility to each possible infinite output string, in such a way that we attach the largest numeric utility to the specific string that we want. And you are claiming this in the same thread in which you disparage all forms of discounting the future. What am I missing here?

0timtyler14y

For my usage, see: * http://carg2.epfl.ch/Publications/2004/PhysicaD04-Mange.pdf The term has subsequently become overloaded, it is true. If I understand it correctly, the rest of your comment is a quibble about infinity. I don't "get" that. Why not just take things one output symbol at a time?

0Perplexed14y

Wow. I didn't see that one coming. Self-reproducing cellular automata. Brings back memories. Well, it wasn't just a quibble about infinity. There was also the dig about discount rates. ;) But I really am mystified. Is a 'step' in this kind of computation to output a symbol and switch to a different state? Are there formulas for calculating utilities? What data go into the calculation? Exactly how does computation work here? Perhaps I need an example. How would you use this 'utility maximization as a programming language' scheme to program the machine to compute the square root of 2? I really don't understand how this is related to either lambda calculus or Turing machines. Why don't you take some time, work out the details, and then produce one of your essays?

0timtyler14y

I didn't (and still don't) understand how discount rates were relevant - if not via considering the comment about infinite output strings. What data go into the calculation of utilities? The available history of sense data, memories, and any current inputs. The agent's internal state, IOW. Just like it normally does? You just write the utility function in a Turing-complete language - which you have to do anyway if you want any generality. The only minor complication is how to get a (single-valued) "function" to output a collection of motor outputs in parallel - but serialisation provides a standard solution to this "problem". Universal action might get an essay one day. ...and yes, if I hear too many more times that humans don't have utility functions (we are better than that!) - or that utility maximisation is a bad implementation plan - I might polish up a page that debunks those - ISTM - terribly-flawed concepts - so I can just refer people to that.

0Perplexed14y

What is it that the agent acts so as to maximize? * The utility of the next action (ignoring the utility of expected future actions) * The utility of the next action plus a discounted expectation of future utilities. * The simple sum of all future expected utilities. To me, only the first two options make mathematical sense, but the first doesn't really make sense as a model of human motivation.

-2timtyler14y

I would usually answer this with a measure of inclusive fitness. However, it appears here that we are just talking about the agent's brain - so in this context what that maximises is just utility - since that is the conventional term for such a maximand. Your options seem to be exploring how agents calculate utilities. Are those all the options? An agent usually calculates utilities associated with its possible actions - and then chooses the action associated with the highest utility. That option doesn't seem to be on the list. It looks a bit like 1 - but that seems to specifiy no lookahead - or no lookahead of a particular kind. Future actions are usually very important influences when choosing the current action. Their utilities are usually pretty important too. If you are trying to make sense of my views in this area, perhaps see the bits about pragmatic and ideal utility functions - here: http://timtyler.org/expected_utility_maximisers/

2shokwave14y

Yes. In fact, 2 strictly contains both 1 and 3, by virtue of setting the discount rate to either 0 or 1. But not strictly as important as the utility of the outcome of the current action. The amount by which future actions are less important than the outcome of the current action, and the methods by which we determine that, are what we mean when we say discount rates.

0timtyler14y

That helps understand the options. I am not sure I had enough info to figure out what you meant before. 1 corresponds to eating chocolate gateau all day and not brushing your teeth - not very realistic as you say. 3 looks like an option containing infinite numbers - and 2 is what all practical agents actually do. However, I don't think this captures what we were talking about. Pragmatic utility functions are necessarily temporally discounted - due to resource limitations and other effects. The issue is more whether ideal utility functions can be expected to be so discounted. I can't think why they should be - and can think of several reasons why they shouldn't be - which we have already covered. Infinity is surely not a problem - you can just maximise utility over T years and let T increase in an unbounded fashion. The uncertainty principle limits the predictions of embedded agents in practice - so T won't ever become too large to deal with.

0Perplexed14y

My understanding is that "pragmatic utility functions" are supposed to be approximations to "ideal utility functions" - preferable only because the "pragmatic" are effectively computable whereas the ideal are not. Our argument is that we see nothing constraining ideal utility functions to be finite unless you allow discounting at the ideal level. And if ideal utilities are infinite, then pragmatic utilities that approximate them must be infinite too. And comparison of infinite utilities in the hope of detecting finite differences cannot usefully guide choice. Hence, we believe that discounting at the ideal level is inevitable. Particularly if we are talking about potentially immortal agents (or mortal agents who care about an infinite future). Your last paragraph made no sense. Are you claiming that the consequence of actions made today must inevitably have negligible effect upon the distant future? A rather fatalistic stance to find in a forum dealing with existential risk. And not particularly realistic, either.

-1timtyler14y

You seem obsessed with infinity :-( What about the universal heat death? Forget about infinity - just consider whether we want to discount on a scale of 1 year, 10 years, 100 years, 1,000 years, 10,000 years - or whatever. I think "ideal" short-term discounting is potentially problematical. Once we are out to discounting on a billion year timescale, that is well into the "how many angels dance on the head of a pin" territory - from my perspective. Some of the causes of instrumental discounting look very difficult to overcome - even for a superintelligence. The future naturally gets discounted to the extent that you can't predict and control it - and many phenomena (e.g. the weather) are very challenging to predict very far into the future - unless you can bring them actively under your control. No, The idea was that predicting those consequences is often hard - and it gets harder the further out you go. Long term predictions thus often don't add much to what short-term ones give you.

0shokwave14y

Flippantly: we're going to have billions of years to find a solution to that problem.

0timtyler14y

You have an alternative model of arbitrary computable agents to propose? You don't think the ability to model an arbitrary computable agent is useful? What is the problem here? Surely a simple utility-based framework for modelling the computable agent of your choice is an obvious Good Thing.

2Perplexed14y

I see no problem modeling computable agents without even mentioning "utility". I don't yet see how modeling them as irrational utility maximizers is useful, since a non-utility-based approach will probably be simpler.

0timtyler14y

Similarly, you can model serial computers without mentioning Turing machines and parallel computers without mentioning cellular automata. Yet in those cases, the general abstraction turns out to be a useful and important concept. I think this is just the same.

0timtyler14y

Part of the case for using a utility maximization framework is that we can see that many agents naturally use an internal representation of utility. This is true for companies, and other "economic" actors. It is true to some extent for animal brains - and it is true for many of the synthetic artificial agents that have been constructed. Since so many agents are naturally utility-based, that makes the framework an obvious modelling medium for intelligent agents.

0Jack14y

Is this really true? My understanding is that Deep Blue's position evaluation function was determined by an analysis of a hundreds of thousands of games. Presumably it ranked openings which had a tendency to produce more promotion opportunities higher than openings which tended to produce fewer promotion opportunities (all else being equal and assuming promoting pawns correlates with wins).

0timtyler14y

I wasn't talking about that - I meant it doesn't evaluate board positions with promoted pawns at the start of the game - even though these are common positions in complete chess games. Anyway, forget that example if you don't like it, the point it illustrates is unchanged.

1timtyler14y

The "C" in "CEV" stands for "Coherent". The concept refers to techniques of combining the wills of a bunch of agents. The idea is not normally applied to a population consisting of single human. That would just be EV. I am not aware of any evidence that Yu-El thinks that EV is independent of the .

2nshepperd14y

A CEV optimizer is less likely to do horrific things while its ability to extrapolate volition is "weak". If it can't extrapolate far from the unwise preferences people have now with the resources it has, it will notice that the EV varies a lot among the population, and take no action. Or if the extrapolation system has a bug in it, this will hopefully show up as well. So coherence is a kind of "sanity test". That's one reason that leaps to mind anyway. Of course the other is that there is no evidence any single human is Friendly anyway, so cooperation would be impossible among EV maximizing AI researchers. As such, an AI that maximizes EV is out of the question already. CEV is the next best thing.

3Vladimir_Nesov14y

The argument seems to be, if Preference1 is too different from Preference1, then Preference1 is a bad method of preference-extraction and should be rethought. A good method Preference2 for preference-extraction should have Preference2 much closer to Preference2. And since Preference1 is inadequate, as demonstrated by this test case, Preference1 is also probably hugely worse for cousin_it than Preference2, even if Preference2 is better than Preference2. We are not that wise in the sense that any moral progress we've achieved, if it's indeed progress (so that on reflection, both past and future would agree that the direction was right) and not arbitrary change, shouldn't be a problem for an AI to repeat, and thus this progress in particular (as opposed to other possible differences) shouldn't contribute to differences in extracted preference.

0Eugine_Nier14y

Of course the above constraint isn't nearly enough to uniquely specify Preference2.

[-]steven16y00

Just to check, surely you're not saying an extrapolated version of Archimedes and a thousand people who agreed with him wouldn't have turned out OK?

It seems to me that we have some quite strong evidence against rotten bastards theory in that intelligent and well-informed people IRL seem to converge away from bastardly beliefs. Still, rotten bastards theory seems worth thinking about for negative-utilitarian reasons.

[-]Tim_Tyler16y10

The basic ev-bio necessity behind the psychological unity of human brains is not widely understood.

"Necessity" may be over-stating it. Humans are not very diverse, due to recent genetic bottlenecks. We do have enormous psychological differences between us - due to neotony-induced developmental plasticity - and because there are many ways to break a human. We haven't yet lived in the human hive long enough for the developmental plasticity to result in clearly-discrete queen/worker/warrior castes, with associated mental adaptations, though.

Go ba... (read more)

[-]Tim_Tyler16y00

Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world

I doubt there would have been much controversy. The fate of humanity is bound to be obliteration by far more advanced technology. The idea that our crappy evolved slug bodies, with their sluggish meat brains might go the distance would have seemed pretty ridiculous, even back then.

[-]Roko16y00

@Eliezer: "I'm sure that Archimedes of Syracuse thought that Syracuse had lots of incredibly important philosophical and cultural differences with the Romans who were attacking his city. Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world - due to all those incredibly important philosophical differences."

- I don't see how adding the romans to the CEV algorithm would have made much of a... (read more)

[-]Larry_D'Anna16y00

Virge: Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.

well because we're trying to characterize the sort of psychological diversity that can exist within our species. And this psychological unity argument is saying "we're all the same, except for a mix of one-step changes". This means that any complex adaptation in any human is in almost a... (read more)

[-]Doug_S.16y20

Many people seem to have set their priors to 1 on several facts. I suspect even MiniLuv would have a hard time getting these guys to believe that Christianity is false.

In other words:

Bob: "Abortion is wrong."
Sally: "Do you think that this is something of which most humans ought to be persuadable?"
Bob: "Yes, I do. Do you think abortion is right?"
Sally: "Yes, I do. And I don't think that's because I'm a psychopath by common human standards. I think most humans would come to agree with me, if they knew the facts I

... (read more)

[-]Jadagul16y00

Doug raises another good point. Related to what I said earlier, I think people really do functionally have prior probability=1 on some propositions. Or act as if they do. If "The Bible is the inerrant word of God" is a core part of your worldview, it is literally impossible for me to convince you this is false, because you use this belief to interpret any facts I present to you. Eliezer has commented before that you can rationalize just about anything; if "God exists" or "The Flying Spaghetti Monster exists" or "reincarnation exists" is part of the machinery you use to interpret your experience, in a deep enough way, your experiences can't disprove it.

[-]Eliezer Yudkowsky16y60

If you had the Pope in a holodeck that you completely controlled, do you think you could convince him that Jesus was never resurrected?

Oh hell yeah. Then again, I'd also go for it if you offered me 24 hours. Then again, the Pope almost certainly already believes that Jesus was never resurrected "except in our hearts" or some such, which makes the job much harder.

[-]Doug_S.16y00

If the Pope knows that he's in a holodeck, and that you are controlling it, would that change your answer?

[-]Tim_Tyler16y30

The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.

That argument is not right - as fig wasps demonstrate. Organisms can exhibit developmental plasticity, and show different t... (read more)

[-]Roko16y00

Doug S: "A human's beliefs depend on the order in which he hears arguments."

- indeed. Anyone who has spent any time arguing with Christians should know this. And the effect is auto-catalytic - our beliefs about both facts and values tend to self-reinforce; c.f. "affective death spirals".

As I have claimed earlier in the comments, many issues in the modern world are not issues that our in-built evolutionary urges or "yuck factors" can advise us on directly. So human beliefs on issues such as which politics is best, whether to ... (read more)

[-]Recovering_irrationalist16y00

Eliezer: The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI.

I'd feel a lot safer if you'd extend this back at least to the infanticidal hunter-gatherers, and preferably to apes fighting around the 2001 monolith.

[-]Eliezer Yudkowsky16y20

Recovering, if I write CEV then it ought to work on hunter-gatherers, but the hunter-gatherers could not plausible have understood the concepts involved in CEV (whereas I might be willing to try to explain it to Archimedes, if not any other Syracusans). So "think of a strategy such that it would work for hunter-gatherers" fails here; I can't visualize that counterfactual unless the hunter-gatherers have the concept of math.

The apes fighting around the monolith are outside my moral reference frame and a CEV focused on them would produce different results.

[-]steven16y00

Roko, it doesn't just edit incorrect factual beliefs, there's also the "what if we thought faster, were smarter, were more like the people we wished we were", etc.

[-]Roko16y00

Steven: "what if we thought faster, were smarter, were more like the people we wished we were"

- yes, I'm aware of this, but the first two act in essentially the same way - they cause simulees to more quickly come to factually correct beliefs, and the last is just a "consistent under reflection" condition.

These conditions make little difference to my concern: the algorithm will end up mixing my values (which I like) with values I hate (religious dogma, sharia law, Christian fundamentalism, the naturalistic fallacy/bioluddism ... ), where my beliefs recieve a very small weighting, and those that I dislike receive a very large weighting.

[-]steven16y00

If you think an IQ-200 extreme-polymath pope who pondered all current and future arguments for atheism would still remain a Christian, then maybe.

[-]Roko16y10

Steven: If you think an IQ-200 extreme-polymath pope who pondered all current and future arguments for atheism would still remain a Christian, then maybe.

Unfortunately, I know from personal experience that there are people who are more intelligent than I am who have been "hooked" by religious memes. I know of several Cambridge mathematicians who are currently at PhD level, one of them in mathematical logic, who are members of this evangelical Christian organization. One in particular is extremely bright: triple first, British mathematics Olympiad... (read more)

[-]Tim_Tyler16y-20

If there are smart Christians out there, then where are their works?

The nearest I thing I ever saw was probably Philip Johnson - and he's not that smart.

2Peterdjones13y

There's plenty in history, eg Newton.

[-]Constant216y20

Time - Philip Johnson is not just a Christian but a creationist. Do you mean, "if there are smart creationists out there..."? I don't really pay much attention to the religious beliefs of the smartest mathematicians and scientists and I'm not especially keen on looking into it now, but I would be surprised if all top scientists without exception were atheists. This page seems to suggest that many of the best scientists are something other than atheist, many of those Christian.

[-]steven16y10

PhD in mathematical logic != extreme polymath. I guess Dyson and Tipler come closest to refuting my position, but I wouldn't expect their beliefs to remain constant under a few thousand years of pondering the overcoming bias archives, say.

[-]Caledonian216y00

You can bring a brain to data, but you can't make it think.

Things like an IQ of 200 indicate that a person has certain cognitive strengths. They DO NOT indicate that those strengths will be utilized. Someone who is not concerned about being dishonest with themselves can self-convince of whatever they please.

At this point, there are no rational grounds for a person to accept the truth claims of, say, the various Christian doctrinal groups. A person who accepts them regardless has already decided to suspend rationality - unless a desire for honesty greate... (read more)

[-]Tim_Tyler16y00

What I mean is, if there are smart Christians out there, why can't they put together a coherent argument favouring Christianity?

I mean, is this really the best they can do?

[-]frelkins16y10

@Tim Tyler

"a coherent argument favouring Christianity"

Catholic people believe that they have such a thing, resting on strong philosophical definitions of ontology and truth. Traditionally I think that if one was truly interested in hearing a coherent argument for Christian belief, the Jesuit order specialized in teaching and expounding the philosophy. Short a long session with a Jesuit, you might consult the Catholic Encyclopedia for Christian arguments.

While perhaps you will ultimately agree that their system is coherent, that is, "marked ... (read more)

[-]Jadagul16y00

Caledonian and Tim Tyler: there are lots of coherent defenses of Christianity. It's just that many of them rest on statements like, "if Occam's Razor comes into conflict with Revealed Truth, we must privilege the latter over the former." This isn't incoherent; it's just wrong. At least from our perspective. Which is the point I've been trying to make. They'd say the same thing about us.

Roko: I sent you an email.

[-]Peterdjones13y-20

Well, it is not logically necessary that we have a genuine disagreement. We might be mistaken in believing ourselves to mean the same thing by the words right and wrong, since neither of us can introspectively report our own moral reference frames or unfold them fully."

I think the idea of associating the meaning of a word with a detailed theory of fine-grained set of criteria, allowing you to apply the term in all cases, has disadvantages.

Newtonian theory has a different set of fine grained criteria about gravity than relativistic theory. If we ta... (read more)

Moderation Log