There needs to be a separate word for that subset of our values that is interpersonal, prosocial, to some extent expected to be agreed-upon, which subset does not always win out in the weighing; this subset is often also called "morality" but that would be confusing.

Are you maybe referring to manners/etiquette/propriety?

Eliezer: This actually kinda sounds (almost) like something I'd been thinking for a while, except that your version added one (well, many actually, but the one is one that's useful in getting it to all add back up to normality) "dang, I should have thought" insight.

But I'm not sure if these are equivalent. Is this more or less what you were saying: "When we're talking about 'shouldness', we mean something, or at least we think we mean something. It's not something we can fully explicitly articulate, but if we could somehow fully utterly comp... (read more)

Eliezer, it's a pleasure to see you arrive at this point. With an effective understanding of the subjective/objective aspects supporting a realistic metaethics, I look forward to your continued progress and contributions in terms of the dynamics of increasingly effective evolutionary (in the broadest sense) development for meaningful growth, promoting a model of(subjective) fine-grained, hierarchical values with increasing coherence over increasing context of meaning-making, implemts principles of (objective) instrumental action increasingly effective ove... (read more)

I second Robin's request that you summarize your positions. It helps other folks organize and think about your ideas.

I'm quite convinced about how you analyze the problem of what morality is and how we should think about it, up until the point about how universally it applies. I'm just not sure that 'humans different shards of god shatter' add up to the same thing across people, a point that I think would become apparent as soon as you started to specify what the huge computation actually WAS.

I would think of the output as not being a yes/no answer, but something akin to 'What percentage of human beings would agree that this was a good outcome, or be able to be thus conv... (read more)

2+2=4 no matter who's measuring. Right, for myself and my family, and right, for you and yours, may not always be the same.

If the child on the tracks were a bully who had been torturing my own child (which actions I had previously been powerless to prevent by any acceptable means afforded by my society, and assuming I had exhausted all reasonable alternatives), it might very well feel right to let the bully be annihilated by the locomotive.

Right is reducible as an aggregation of sympathetic conditioning; affection for a person, attachment to conceptualization or expected or desired course of events, and so on.

Wow, there's a lot of ground to cover. For everyone who hasn't read Eliezer's previous writings, he talks about something very similar in Creating Friendly Artificial Intelligence, all the way back in 2001 (link = http://www.singinst.org/upload/CFAI/design/structure/external.html). With reference to Andy Wood's comment:

"What claim could any person or group have to landing closer to the one-place function?"

Next obvious question: For purposes of Friendly AI, and for correcting mistaken intuitions, how do we approximate the rightness function? How d... (read more)

This little accident of the Gift doesn't seem like a good reason to throw away the Gift We've been "gifted" with impulses to replicate our genes, but many of us elect not to. I'm not as old as Steven Pinker is when he seemingly bragged of it, but I've made no progress toward reproducing and don't have any plans for it in the immediate future, though I could easily donate to a sperm bank. I could engage in all sorts of fitness lowering activities like attending grad-school, becoming a Jainist monk, engaging in amputative body-modification or commi... (read more)

I found this post a lot more enlightening than the posts that it's a followup to.

TGGP, as far as I understand, Arrow's theorem is an artifact of forcing people to send only ordinal information in voting (and enforcing IIA which throws away that information on the strength of preferences between two alternative which is available from rankings relative to third alternatives). People voting strategically isn't an issue either when you're extrapolating them and reading off their opinions.

"alternative" -> "alternatives"

I think lots of people are misunderstanding the "1-place function" bit. It even took me a bit to understand, and I'm familiar with the functional programming roots of the analogy. The idea is that the "1-place morality" is a closure over (i.e. reference to) the 2-place function with arguments "person, situation" that implicitly includes the "person" argument. The 1-place function that you use references yourself. So the "1-place function" is one's subjective morality, and not some objective version. I think that could have been a lot clearer in the post. Not everyone has studied Lisp, Scheme, or Haskell.

Overall I'm a bit disappointed. I thought I was going to learn something. Although you did resolve some confusion I had about the metacircular parts of the reasoning, my conclusions are all the same. Perhaps if I were programming an FAI the explicitness of the argument would be impressive.

As other commenters have brought up, your argument doesn't address how your moral function interacts with others' functions, or how we can go about creating a social, shared morality. Granted, it's a topic for another post (or several) but you could at least acknowledge the issue.

Too much rhetoric (wear black, miracle, etc.), you wandered off the point three too many times, you used incoherent examples, you never actually defended realism, you never defend the assertion of "the big computation", and for that much text there was so little actually said. A poor offering.

that subset of our values that is interpersonal, prosocial, to some extent expected to be agreed-upon, which subset does not always win out in the weighing

Can we just say that evolution gave most of us such an identifiable subset, and declare a name for that? Even so, a key question remains whether we are mistaken in expecting agreement - are we built to actually agree given enough analysis and discussion, or only to mistakenly expect to agree?

I agree with Andy Wood and Nick Tarleton. To put what they have said another way, you have taken the 2-place function

Rightness(person,act)

And replaced it with a certain unspecified unary rightness function which I will call "Eliezer's_big_computation( -- )". You have told us informally that we can approximate

Eliezer's_big_computation( X ) = happiness( X ) + survival( X ) + justice( X ) + individuality( X ) + ...

But others may define other "big computations". For example

God's_big_computation( X ) = submission( X ) + Oppression_of_women(... (read more)

Let me see if I get this straight:

Our morality is composed of a big computation that includes a list of the things that we value(love, friendship, happiness,...) and a list of valid moral arguments(contagion backward in time, symmetry,...). If so, then how do we discover those lists? I guess that the only way is to reflect on our own minds, but if we do that, then how do we know if a particular value comes from our big computation, or is it just part of our regular biases? And if our biases are inextricably tangled with The Big Computation, then what hope ... (read more)

I second Behemouth and Nick- what do we do in the mindspace in which individual's feelings of right and wrong disagree? What if some people think retarded children absolutely should NOT be pulled off the track? Also, what about the pastrami-sandwich dilemma? (hat of those who would kill 1 million unknown people with no consequence to themselves for a delicious sandwich?

But generally, I loved the post. You should write another post on 'Adding Up to Normality.'

Just because I can't resist, a poem about human failing, the judgment of others we deem weaker than ourselves, and the desire to 'do better.' Can we?

"No Second Troy" WB Yeats, 1916 WHY should I blame her that she filled my days With misery, or that she would of late Have taught to ignorant men most violent ways, Or hurled the little streets upon the great, Had they but courage equal to desire? 5 What could have made her peaceful with a mind That nobleness made simple as a fire, With beauty like a tightened bow, a kind That is not natural in an age like this, Being high and solitary and most stern? 10 Why, what could she have done being what she is? Was there another Troy for her to burn?

P.S : My great "Aha!" moment from reading this post is the realisation that morality is not just a utility function that maps states of the world to real numbers, but also a set of intuitions for changing that utility function.

Added a section on non-universalizability:

If you hoped that morality would be universalizable - sorry, that one I really can't give back. Well, unless we're just talking about humans. Between neurologically intact humans, there is indeed much cause to hope for overlap and coherence; and a great and reasonable doubt as to whether any present disagreement is really unresolvable, even it seems to be about "values". The obvious reason for hope is the psychological unity of humankind, and the intuitions of symmetry, universalizability, and simplic

Dear Eliezer, First of all, great post, thank you, I truly love you Eli!! It was really the kind of beautiful endpoint in your dance I was waiting for, and it is very much in the lines of my own reasoning, just a lot more detailed. I also think this could be labeled metametamorality, therefore some of the justified complaints does not yet apply. But the people complaining about different moral preferences are doing so with their own morality, what else could they be using, and in doing so they are acting according to the arguments of this post. Metametamor... (read more)

P.S. So my additinon is really, choose a stable value structure that feels right, try to maximize it, try to make it better and change so when you feel it is right. I have my own high-level suggestion of Beauty, Truth and the Good, and I later discovered Plato and a lot of others seem to argue for the same three...

There are some good thoughts here, but I don't think the story is a correct and complete account of metamorality (or as the rest of the world calls it: metaethics). I imagine that there will be more posts on Eliezer's theory later and more opportunities to voice concerns, but for now I just want to take issue with the account of 'shouldness' flowing back through the causal links.

'Shouldness' doesn't always flow backwards in the way Eliezer mentioned. e.g. Suppose that in order to push the button, you need to shoot someone who will fall down on it. This wou... (read more)

On reflection, there should be a separate name for the space of arguments that change our terminal values. Using "metaethics" to indicate beliefs about the nature of (ontology of) morality, would free up "metamorals" to indicate those arguments that change our terminal values. So I bow to Zubon and standard usage - even though it still sounds wrong to me.

Toby, the case of needing to shoot someone who will fall down on the button, is of course very easy for a consequentialist to handle; wrongness flows backward from the shooting, as ri... (read more)

Suggested summary: "There is nothing else." That is the key sentence. After much discussion of morals and metas, it comes down to: "You go on with the same morals as before, and the same moral arguments as before." The insight offered is that there is no deeper insight to offer. The recursion will bottom out, so bite the bullet and move on.

Yet another agreement on the 1-Place and 2-Place problem, and I read it after the addition. CEV goes around most of that for neurologically intact humans, but the principle of "no universall... (read more)

wrongness flows backward from the shooting, as rightness flows backward from the button, and the wrongness outweighs the rightness.

I suppose you could say this, but if I understand you correctly, then it goes against common usage. Usually those who study ethics would say that rightness is not the type of thing that can add with wrongness to get net wrongness (or net rightness for that matter). That is, if they were talking about that kind of thing, they wouldn't use the word 'rightness'. The same goes for 'should' or 'ought'. Terms used for this kind of st... (read more)

Eliezer: "if you were stepping outside the human and hoping for moral arguments that would persuade any possible mind, even a mind that just wanted to maximize the number of paperclips in the universe, then sorry - the space of possible mind designs is too large to permit universally compelling arguments."

• I disagree.

After thinking more about it, I might be wrong: actually the calculation might end up giving the same result for every human being.

Caledonian: what kind of motivations do you have?

Okay, for the future I'll just delete the content-free parts of Caledonian's posts, like those above. There do seem to be many readers who would prefer that he not be banned outright. But given the otherwise high quality of the comments on Overcoming Bias, I really don't think it's a good idea to let him go on throwing up on the blog.

Watching the ensuing commentary, I'm drawn to wishfully imagine a highly advanced Musashi, wielding his high-dimensional blade of rationality such that in one stroke he delineates and separates the surrounding confusion from the nascent clarity. Of course no such vorpal katana could exist, for if it did, it would serve only to better clear the way for its successors.

I see a preponderance of viewpoints representing, in effect, the belief that "this is all well and good, but how will this guide me to the one true prior, from which Archimedian point one ... (read more)

Caledonian: uh... he didn't say you couldn't make arguments about all possible minds, he was saying you couldn't construct an argument that's so persuasive, so convincing that every possible mind, no matter how unusual its nature, would automatically be convinced by that argument.

It's not a matter of talking about minds, it's a matter of talking to minds.

Mathematicians figure out things about sets. But they're not trying to convince the sets themselves about those things. :)

Roko: You think you can convince a paperclip maximizer to value human life? Or do you think paperclip maximizers are impossible?

Eliezer: It's because when I say right, I am referring to a 1-place function

Like many others, I fall over at this point. I understand that Morality_8472 has a definite meaning, and therefore it's a matter of objective fact whether any act is right or wrong according to that morality. The problem is why we should choose it over Morality_11283.

Of course you can say, "according to Morality_8472, Morality_8472 is correct" but that's hardly helpful.

Ultimately, I think you've given us another type of anti-realist relativism.

Eliezer: But if you were ste... (read more)

Caledonian: He isn't using "too-big" in the way you are interpreting it.

The point is not: Mindspace has a size X, X > Y, and any set of minds of size > Y cannot admit universal arguments.

The point is: For any putative universal argument you can cook up, I can cook up a mind design that isn't convinced by it.

The reason that we say it is too big is because there are subsets of Mindspace that do admit universally compelling arguments, such as (we hope) neurologically intact humans.

Thank you for this post. "should" being a label for results of the human planning algorithm in backward-chaining mode the same way that "could" is a label for results of the forward-chaining mode explains a lot. It's obvious in retrospect (and unfortunately, only in retrospect) to me that the human brain would do both kinds of search in parallel; in big search spaces, the computational advantages are too big not to do it.

I found two minor syntax errors in the post: "Could make sense to ..." - did you mean "Could it make s... (read more)

Suppose we were to write down all (input, output) pairs for the ideal "one-place function" described by Eliezer on a oblong stone tablet somewhere. This stone tablet would then contain perfect moral wisdom. It would tell us the right course of action in any possible situation.

This tablet would be the result of computation, but it's computation that nobody can actually do, as we currently only have access to approximations to the ideal Morality(X) function. Thus, as far as we're concerned, this tablet is just a giant look-up table. Its content... (read more)

Calhedonian: [THIS WOULD GET DELETED]The reason you are unable to make such arguments is that you're unwilling to do any of the rudimentary tasks necessary to do so. You've accomplished nothing but making up names for ill-defined ideas and then acting as though you'd made a breakthrough. On the off-chance that you actually want to contribute something meaningful to the future of humanity, I suggest you take a good, hard look at your other motivations - and the gap between what you've actually accomplished and your espoused goals.[/THIS WOULD GET DELETED]

Th... (read more)

I'm still wrestling with this here -

Do you claim that the CEV of a pygmy father would assert that his daughter's clitoris should not be sliced off? Or that the CEV of a petty thief would assert that he should not possess my iPod?

Mike Blume: Do you claim that the CEV of a pygmy father would assert that his daughter's clitoris should not be sliced off? Or that the CEV of a petty thief would assert that he should not possess my iPod?

Mike, a coherent extrapolated volition is generally something you do with more than one extrapolated volition at once, though I suppose you could extrapolate a single human's volition into a spread of outcomes and look for coherence in the spread. But this level of metaethics is of interest primarily to FAIfolk, I would think.

With that said, if I were b... (read more)

Unknown: "But it is quite impossible that the complicated calculation in Eliezer's brain should be exactly the same as the one in any of us: and so by our standards, Eliezer's morality is immoral. And this opinion is subjectively objective, i.e. his morality is immoral and would be even if all of us disagreed. So we are all morally obliged to prevent him from inflicting his immoral AI on us"

Well, I would agree with this point if I thought what Eliezer was going to inflict upon us was so out of line with what I want that we would be better off wit... (read more)

Unknown wrote:
As I've stated before, we are all morally obliged to prevent Eliezer from programming an AI.

As Bayesians, educated by Mr. Yudkowsky himself, I think we all know the probability of such an event is quite low. In 2004, in the most moving and intelligent eulogy I have ever read, Mr. Y stated: "When Michael Wilson heard the news, he said: "We shall have to work faster." Any similar condolences are welcome. Other condolences are not." Somewhere, some person or group is working faster, but at the Singularity Institute, all the... (read more)

Any chance of a post summarizing all of the building block posts for this topic, like you did with your physics posts?
I hate to be a beggar, but that would be very helpful.

Just another point as to why important, meglomeniacal types like Eliezer need to have their motives checked:
Frank Vertosick, in his book "When the Air Hits Your Brain: Tales from Neurosurgery," about a profession I am seriously considering, describes what becomes of nearly all people taking such power over life and death:

"He was the master... the 'ptototypical surgical psychopath' - someone who could render a patient quadriplegic in the morning, play golf in the afternoon, and spend the evening fretting about that terrible slice off the sev... (read more)

About the comments on compromise: that's why I changed my mind. The functions are so complex that they are bound to be different in the complex portions, but they also have simplifying terms in favor of compromise, so it is possible that everyone's morality will end up the same when this is taken into account.

As for the probability that Eliezer will program an AI, it might not be very low, but it is extremely low that his will be the first, simply because so many other people are trying.

I'm near Unknown's position. I don't trust any human being with too much power. No matter how nice they seem at first, history indicates to me that they inevitably abuse it. We've been told that a General AI will have power beyond any despot known to history. Am I supposed to have that much reliance on the essential goodness within Eliezer's heart? And in case anyone brings this up, I certainly don't trust the tyranny of the majority either. I don't recognize any moral obligation to stop it because I don't recognize any obligations at all. Also, I might no... (read more)

Larry: You think you can convince a paperclip maximizer to value human life? Or do you think paperclip maximizers are impossible?

I don't think that convincing arbitrary minds is the point. The point is that there does exist a canonical system of values. Just because there's an objectively true and canonical morality doesn't mean that every mind in existence can be persuaded to follow it: some minds are simply not rational.

Perhaps I should not have said "I disagree" to Eli, I should have said "what you say is trivially true, but it misses the point"

Female circumcision is commonly carried out by women who've undergone the procedure themselves.

Then they don't know the true difference between the two possible lives, do they?

I see this line of thinking coming directly out of Hume. Some of Hume's main points, as I read him:

1. Morality flows straight from humanity's values, and that's it.

2. Morality is universalizable among humans because of the psychological commonality.

3. What we are really doing in ethics is trying to find general principles which explain the values we have, then we can use the general principle to make ethical decisions. This is another label for trying to define the big abstract computation in our heads so that we can better optimize it. Hume never re... (read more)

I don't understand why it must be a given that things like love, truth, beauty, murder, etc.. are universal moral truths that are right or wrong independent of the person computing the morality function. I know you frown upon mentioning evolutionary psychology, but is it really a huge stretch to surmise that the more even-keeled, loving and peaceful tribes of our ancestors would out-survive the wilder warmongers who killed each other out? Even if their good behavior was not genetic, the more "moral" leaders would teach/impart their morality to th... (read more)

TGGP:
I have great sympathy with this position. An incorrectly formatted AI is one of the biggest fears of the singularity institute, mainly because there are so many more ways to be way wrong than even slightly right about it... It might be that the task of making an actually friendly AI is just too difficult for anyone, and our efforts should be spent in preventing anyone from creating a generally intelligent AI, in the mean time trying to figure out, with our inperfect human brains and the crude tools at our disposal, how to make uploads ourselves or c... (read more)

Matt Simpson: Many an experiment has been thought for the sole purpose of showing how utilitarianism is in direct conflict with our moral intuitions.

I disagree, or you're referring to something I haven't heard of. If I know what you mean here, those are a species of strawman ("act") utilitarianism that doesn't account for the long-term impact and adjustment of behavior that results.

(I'm going to stop giving the caveats; just remember that I accept the possibility you're referring to something else.)

For example, if you're thinking about cases whe... (read more)

So, what do you do about inter-morality? Suppose a paperclip maximizer with the intellect and self-improvement limits of a typical human thirteen-year-old. They are slightly less powerful than you and can be expected to stay that way, so they aren't an existential threat. They are communicative and rational but they have one terminal value, and it's paperclips.

How ought a moral human to react? Can you expect to negotiate an inter-morality? Or must the stronger party win by force and make the weaker party suffer non-fulfillment? Is this mis-treatment and immoral? Ought you to allow them a bale of wire so they can at least make some paperclips?

I second TGGP that the mention of wearing black etc. is ridiculous.

Lara: The portion you quote from Caledonian isn't at all well-defined itself; it's a near-pure insult hinting at, but not giving, actual arguments. I fully support its deletion. Also, Eliezer isn't saying "keep on believing what you believe", but "keep on following the process you have been"; he allows for moral error.

Lara, TGGP: The most important point is that building an AI, unlike surgery or dictatorship, doesn't give you any power to be corrupted by - any opportunit... (read more)

Eliezer [in response to me]: This just amounts to defining should as an abstract computation, and then excluding all minds that calculate a different rule-of-action as "choosing based on something other than morality". In what sense is the morality objective, besides the several senses I've already defined, if it doesn't persuade a paperclip maximizer?

I think my position is this:

If there really was such a thing as an objective morality, it would be the case that only a subset of possible minds could actually discover or be persuaded of that fact.

Presumably, for any objective fact, there are possible minds who could never be convinced of that fact.

Caledonian: uh... he didn't say you couldn't make arguments about all possible minds, he was saying you couldn't construct an argument that's so persuasive, so convincing that every possible mind, no matter how unusual its nature, would automatically be convinced by that argument.

That point is utterly trivial. You can implement any possible relationship between input and output. That even includes minds that are generally rational but will fail only in specifically-defined instances - such as a particular person making a particular argument. This does... (read more)

Caledonian: I can't think of anyone EVER choosing to interpret statements as stupid rather than sensible to the degree to which you do on this blog. There is usually NO ambiguity and you still get things wrong and then blame them for being stupid.

In all honesty why do you post here? On your own blog you are articulate and intelligent. Why not stick with that and leave commenting to people who want to actually respond to what people say rather than to straw men?

Nick,

Eliezer's one-place function is exactly infallible, because he defines "right" as its output.

I misunderstood some of Eliezer's notation. I now take his function to be an extrapolation of his volition rather than anyone else's. I don't think this weakens my point: if there were a rock somewhere with a lookup table for this function written on it, Eliezer should always follow the rock rather than his own insights (and according to Eliezer everyone else should too), and this remains true even if there is no such rock.

Furthermore, the morality... (read more)

I can't think of anyone EVER choosing to interpret statements as stupid rather than sensible to the degree to which you do on this blog.

[MISREPRESENTATION]If, as Eliezer's defenders insist, we should interpret his remarks as suggesting that there is no point to looking for convincing arguments regarding 'morality' because there are no arguments that will convince all possible minds,[/MISREPRESENTATION] how exactly can this be construed as sensible? How is this compatible w... (read more)

I don't know much about the ins-and-outs of blog identification. Is it possible that someone could post diametrically, under two names, such as Caledonian and Robin Hanson, in order to maintain a friendship while minimizing the importance of efforts that one considers unimportant?

Constant: Corruptibility is a complex evolutionary adaptation. Even the best humans have hard-to-suppress semiconscious selfish motivations that tend to come out when we have power, even if we thought we were gaining that power for idealistic reasons. There's no reason an AI with an intelligently designed, clean, transparent goal system would be corrupted, or need to be kept in check. "Respect for rights" is similarly anthropomorphic. Creating a society of AIs would be very problematic due to the strong first-mover effect, and the likely outcome ... (read more)

Why not, I can't help myself: Caledonian = Thersites, Eliezer = Agamemnon

Thersites only clamour’d in the throng,
Loquacious, loud, and turbulent of tongue:
Awed by no shame, by no respect controll’d,
In scandal busy, in reproaches bold:
With witty malice studious to defame,
Scorn all his joy, and laughter all his aim:—
But chief he gloried with licentious style
To lash the great, and monarchs to revile.
...
Sharp was his voice; which in the shrillest tone,
Thus with injurious taunts attack’d the throne.
Whate’er our master craves submit we must,... (read more)

This post is called the "The Meaning of Right", but it doesn't spend much time actually defining what situations should be considered as right instead of wrong, other than a bit at the end which seems to define "right" as simply "happiness". Rather its a lesson in describing how to take your preferred world state, and causally link that to what you'd have to do to get to that state. But that world state is still ambiguously right/wrong, according to any absolute sense, as of this post.

So does this post say what "right" means, other than simply "happiness" (which sounds like generic utilitarianism), am I simply missing something?

Eliezer once wrote that "We can build up whole networks of beliefs that are connected only to each other - call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens's ability to build more general and flexible belief networks.

The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit."

I can't see how nearly all of the beliefs expressed in this post predict or prohibit any experience.

Silas: I'm referring to all thought experiments where the intended purpose was to show that utilitarianism is inconsistent with our moral intuitions. So, yes, the examples you mention, and more. Most of them do fall short of their purpose.

Nick Tarleton: I'm not sure all seemingly anti-utilitarian intuitions can be explained away by scope insensitivity, but that does take care of the vast majority of cases.

One case I was thinking of (for both of you) is the 'utility monster:' someone who receives such glee from killing, maiming, and otherwise causing ... (read more)

Then they don't know the true difference between the two possible lives, do they?
"True difference" gets me thinking of "no true Scotsman". Has there ever been anybody who truly knew the difference between two possible lives? Even if someone could be reincarnated and retain memories the order would likely alter their perceptions.

I'm very interested in how Eliezer gets from his meta-ethics to utilitarianism
He's not a strict utilitarian in the "happiness alone" sense. He has an aversion to wireheading, which maximizes the class... (read more)

It seems to me you're saying that what our conscience tells us is right is right because it's output is what we mean by "right" in the first place. While I agree in general that a concept is it's referents, I don't agree with what you're saying here.

Those referents are not values but evaluations. And they are evaluations with respect to a standard that we can in fact change. We don't choose the output of our conscience on the spot, in that sense it is objective, but over time we can reprogram it through repetition and effort. It's evaluations are short-term objective but long term subjective.

Just do what is, ahem, right - to the best of your ability to weigh the arguments you have heard, and ponder the arguments you may not have heard.

What's "the best of your ability"? 'Best' is a determination of quality. What constitutes quality reasoning about 'morality'?

When we talk about quality reasoning in, say, math, we don't have problems with that question. We don't permit just any old argument to be acceptable - if people's reasoning doesn't fit certain criteria, we don't accept that reasoning as valid. That they make the arguments, ... (read more)

I'm confused. I'll try to rephrase what you said, so that you can tell me whether I understood.

"You can change your morality. In fact, you do it all the time, when you are persuaded by arguments that appeal to other parts of your morality. So you may try to find the morality you really should have. But - "should"? That's judged by your current morality, which you can't expect to improve by changing it (you expect a particular change would improve it, but you can't tell in what direction). Just like you can't expect to win more by changing yo... (read more)

TGGP wrote:

We've been told that a General AI will have power beyond any despot known to history.

If that will be then we are doomed. Power corrupts. In theory an AI, not being human, might resist the corruption, but I wouldn't bet on that. I do not think it is a mere peculiarity of humanity that we are vulnerable to corruption.

A mind with access to its source code, if it doesn't want to be corrupted, won't be.

What does 'corrupted' mean in this context?

If we go by definitions, we have

6. to destroy the integrity of; cause to be dishonest, disloyal, etc., esp. by bribery.
7. to lower morally; pervert: to corrupt youth.
8. to alter (a language, text, etc.) for the worse; debase.
9. to mar; spoil.
10. to infect; taint.
11. to make putrid or putrescent.

Most of those meanings cannot apply here - and the ones that do refer to changes in morality.

A tendency to become corrupt when placed into positions of power is a feature of some minds.

Morality in the human universe is a compromise between conflicting wills. The compromise is useful because the alternative is conflict, and conflict is wasteful. Law is a specific instance of this, so let us look at property rights: property rights is a decision-making procedure for deciding between conflicting desires concerning the owned object. There really is no point in even having property rights except in the context of the potential for conflict. Remove conf... (read more)

I am assuming [the AI] acts, and therefore makes choices, and therefore has preferences, and therefore can have preferences which conflict with the preferences of other minds (including human minds).

An AI can indeed have preferences that conflict with human preferences, but if it doesn't start out with such preferences, it's unclear how it comes to have them later.

On the other hand, if it starts out with dubious preferences, we're in trouble from the outset.

Constant: "Give a person power, and he no longer needs to compromise with others, and so for him the raison d'etre of morality vanishes and he acts as he pleases."

If you could do so easily and with complete impunity, would you organize fights to death for your pleasure? Would you even want to? Moreover, humans are often tempted to do things they know they shouldn't, because they also have selfish desires. AIs don't if you don't build it into them. If they really do ultimately care about humanity's well-being, and don't take any pleasure from making people obey them, they will keep doing so.

An AI can indeed have preferences that conflict with human preferences, but if it doesn't start out with such preferences, it's unclear how it comes to have them later.

We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted. For example, babies do not start out preferring Bach to Beethoven or Beethoven to Bach, but adults are able to develop that preference, even if it is not clear at this point how they come to do so.

If you could do so easily ... (read more)

Larry D'Anna: The reason that we say it is too big is because there are subsets of Mindspace that do admit universally compelling arguments, such as (we hope) neurologically intact humans.

What precisely is neurological intactness? It rather seems to me that the majority agrees on some set of "self-evident" terminal values, and those few people that do not are called psychopaths. If by "human" we mean what usually people understand by this term, then there are no compelling arguments even for humans. Althoug I gladly admit your statement is approximatively valid, I am not sure how to formulate it to be exactly true and not simultaneously a tautology.

Constant [sorry for getting the attribution wrong in my previous reply] wrote:

We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted.

Any two AIs are likely to have a much vaster difference in effective intelligence than you could ever find between two humans (for one thing, their hardware might be much more different than any two working human brains). This likelihood increases further if (at least) some subset of them is capable of strong self-improvement. With enough difference in power, cooperation becomes a losing strategy for the more powerful party.

I read stuff like this and immediately my mind thinks, "comparative advantage." The point is that it can be (and probably is... (read more)

Humans having this kind of tendency is a predictable result of what their design was optimized to do, and as such them having it doesn't imply much for minds from a completely different part of mind design space.
Eliezer seems to be saying his FAI will emulate his own mind, assuming it was much more knowledgeable and had heard all the arguments.

Um, no. First, the last revision of the plan called for focusing the FAI on the whole human species, not just one or more programmers. Second, the extrapolation is a bit more complicated than "if you knew more". I am neither evil nor stupid.

After all, if the humans have something worth treating as spoils, then the humans are productive and so might be even more useful alive.

I read stuff like this and immediately my mind thinks, "comparative advantage." The point is that it can be (and probably is) worthwhile for Bob and Bill to t

We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted.

I believe Eliezer is trying to create "fully recursive self-modifying agents that retain stable preferences while rewriting their source code". Like Sebastian says, getting the "stable preferences" bit right is presumably necessary for Friendly AI, as Eliezer sees it.

(This clause "as Eliezer sees it" isn't meant to indicate dissent, but merely my total incompetence to judge whether this condition is strictly necessary for friendly AI.)

I find it interesting that I found many of the posts leading up to this one intensely hard to follow as they seemed to be arguing against worldviews that I had little or no comprehension of.

So, I must say that I am very relieved to see that your take on what morality is, is what I've been assuming it is, all along: just a fascinating piece of our internal planning software.