Eliezer_Yudkowsky comments on A Less Wrong singularity article? - Less Wrong

28 Post author: Kaj_Sotala 17 November 2009 02:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (210)

Tags: singularity ai crowdsourcing

Comments (210)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 19 November 2009 07:26:30PM * 0 points [-]

My use is not counterintuitive. The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.

If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.

Comment author: wedrifid 19 November 2009 07:51:38PM * 5 points [-]

If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.

One of the standard usages is "doing this will most enhance your utility". As in "you should kill that motherf@#$%". This is distinct from 'right' and 'good' although 'ought' is used in the same way, albeit less frequently. It is advice, rather than exhortation.

Comment author: CronoDAS 19 November 2009 07:57:28PM * 7 points [-]

Indeed. "The Pebblesorters should avoid making piles of 1,001 stones" makes perfect sense.

Comment author: timtyler 19 November 2009 08:06:35PM 0 points [-]

"Should" and "ought" actually have strong connotations of societal morality.

Should you rob the bank? Should you have sex with the minor? Should you confess to the crime?

Your personal utility is one thing - but "should" and "ought" often have more to do with what society thinks of your actions.

Comment author: wedrifid 19 November 2009 08:25:24PM 0 points [-]

Should you rob the bank?

Probably not.

Should you have sex with the minor?

Probably not here.

Should you confess to the crime?

Hell no. "The Fifth" is the only significant law-item that I'm explicitly familiar with. And I'm not even American.

Your personal utility is one thing - but "should" and "ought" often have more to do with what society thinks of your actions.

More often what you want society to think of people's actions (either as a signal or as persuasion. I wonder which category my answers above fit into?).

Comment author: timtyler 19 November 2009 07:48:24PM * 5 points [-]

It's counterintuitive to me - and I'm not the only one - if you look at the other comments here.

Aliens could have the "right", "good", "ought" and "should" concept cluster - just as some other social animals can, or other tribes, or humans at other times.

Basically, there are a whole bunch of possible and actual moral frameworks - and these words normally operate relative to the framework under consideration.

There are some people who think that "right" and "wrong" have some kind of universal moral meaning. However most of those people are religious, and think morality comes straight from god - or some such nonsense.

Comment author: Eliezer_Yudkowsky comment score below threshold [+] (19 children)

Comment author: Vladimir_Nesov 19 November 2009 08:36:17PM 6 points [-]

If you are talking about a different concept, you should use a different word.

Not how natural language works.

Comment author: Cyan 19 November 2009 09:20:57PM * 2 points [-]

If you are talking about a different concept, you should use a different word.

Do you mean it would be right and good for him to use a different word, or that it would be more effective communication if he did so?

Comment author: timtyler 19 November 2009 08:22:57PM * 1 point [-]

To clarify, people agree that the moral "right" and "wrong" categories contain things that are moral and immoral respectively - but they disagree with each other about which actions are moral and which are immoral.

For example, some people think abortion is immoral. Other people think eating meat is immoral. Other people think homosexual union is immoral - and so on.

These opinions are not widely agreed upon - yet many of those who hold them defend them passionately.

Comment deleted 19 November 2009 08:36:07PM * [-]

Comment author: Eliezer_Yudkowsky 19 November 2009 08:49:49PM 4 points [-]

Okay, then these are the people I'm arguing against, as a view of morality. I'm arguing that, say, dragging 6-year-olds off the train tracks, as opposed to eating them for lunch, is every bit as much uniquely the right answer as it looks; and that the Space Cannibals are every bit as awful as they look; and that the aliens do not have a different view of the subject, but simply a view of a different subject.

As an intuition pump, it might help to imagine someone saying that "truth" has different values in different places and that we want to parameterize it by true<Islam> and true<Science>. If Islam has a sufficiently different criterion for using the word "true", i.e. "recorded in the Koran", then we just want to say "recorded in the Koran", not use the word "true".

Another way of looking at it is that if we are not allowed to use the word "right" or any of its synonyms, at all, a la Empty Labels and Rationalist Taboo and Replace the Symbol with the Substance, then the new language that we are forced to use will no longer create the illusion that we and the aliens are talking about the same thing. (Like forcing people from two different spiritual traditions to say what they think exists without using the word "God", thus eliminating the illusion of agreement.) And once you realize that we and the aliens are not talking about the same thing at all, and have no disagreement over the same subject, you are no longer tempted to try to relativize morality.

It's all very well to tell me that I should stop arguing over definitions, but I seem to be at a loss to make people understand what I am trying to say here. You are, of course, welcome to tell me that this is my fault; but it is somewhat disconcerting to find everyone saying that they agree with me, while continuing to disagree with each other.

Comment author: Alicorn 19 November 2009 08:56:21PM 2 points [-]

I disagree with you about what "should" means, and I'm not even a Space Cannibal. Or do I? Are you committed to saying that I, too, am talking past you if I type "should" to sincerely refer to things?

Are you basically declaring yourself impossible to disagree with?

Comment author: Eliezer_Yudkowsky 19 November 2009 09:03:02PM 3 points [-]

Do you think we're asking sufficiently different questions such that they would be expected to have different answers in the first place? How could you know?

Humans, especially humans from an Enlightenment tradition, I presume by default to be talking about the same thing as me - we share a lot of motivations and might share even more in the limit of perfect knowledge and perfect reflection. So when we appear to disagree, I assume by default and as a matter of courtesy that we are disagreeing about the answer to the same question or to questions sufficiently similar that they could normally be expected to have almost the same answer. And so we argue, and try to share thoughts.

With aliens, there might be some overlap - or might not; a starfish is pretty different from a mammal, and that's just on Earth. With paperclip maximizers, they are simply not asking our question or anything like that question. And so there is no point in arguing, for there is no disagreement to argue about. It would be like arguing with natural selection. Evolution does not work like you do, and it does not choose actions the way you do, and it was not disagreeing with you about anything when it sentenced you to die of old age. It's not that evolution is a less authoritative source, but that it is not saying anything at all about the morality of aging. Consider how many bioconservatives cannot understand the last sentence; it may help convey why this point is both metaethically important and intuitively difficult.

Comment author: Alicorn 19 November 2009 09:22:52PM * 4 points [-]

Do you think we're asking sufficiently different questions such that they would be expected to have different answers in the first place? How could you know?

I really do not know. Our disagreements on ethics are definitely nontrivial - the structure of consequentialism inspires you to look at a completely different set of sub-questions than the ones I'd use to determine the nature of morality. That might mean that (at least) one of us is taking the wrong tack on a shared question, or that we're asking different basic questions. We will arrive at superficially similar answers much of the time because "appeal to intuition" is considered a legitimate move in ethics and we have some similar intuitions about the kinds of answers we want to arrive at.

I think you are right that paperclip maximizers would not care at all about ethics. Babyeaters, though, seem like they do, and it's not even completely obvious to me that the gulf between me and a babyeater (in methodology, not in result) is larger than the gulf between me and you. It looks to me a bit like you and I get to different parts of city A via bicycle and dirigible respectively, and then the babyeaters get to city B via kayak - yes, we humans have more similar destinations to each other than to the Space Cannibals, but the kind of journey undertaken seems at least as significant, and trying to compare a bike and a blimp and a boat is not a task obviously approachable.

Comment author: Eliezer_Yudkowsky 19 November 2009 09:28:05PM 4 points [-]

I find it a suspicious coincidence that we should arrive at similar answers by asking dissimilar questions.

Comment author: Alicorn 19 November 2009 09:29:58PM * 0 points [-]

Do you also find it suspicious that we could both arrive in the same city using different vehicles? Or that the answer to "how many socks is Alicorn wearing?" and the answer to "what is 6 - 4?" are the same? Or that one could correctly answer "yes" to the question "is there cheese in the fridge?" and the question "is it 4:30?" without meaning to use a completely different, non-yes word in either case?

continue this thread »

Comment author: UnholySmoke 21 November 2009 12:34:11AM 0 points [-]

I think you are right that paperclip maximizers would not care at all about ethics.

Correct. But neither would they 'care' about paperclips, under the way Eliezer's pushing this idea. They would flarb about paperclips, and caring would be as alien to them as flarbing is to you.

Comment author: Alicorn 21 November 2009 12:36:55AM 2 points [-]

I think some subset of paperclip maximizers might be said to care about paperclips. Not, most likely, all possible instances of them.

Comment author: wedrifid 19 November 2009 09:48:26PM -1 points [-]

Babyeaters, though, seem like they do, and it's not even completely obvious to me that the gulf between me and a babyeater (in methodology, not in result) is larger than the gulf between me and you.

I had the same thought.

Comment author: wedrifid 19 November 2009 09:40:55PM * 0 points [-]

It's all very well to tell me that I should stop arguing over definitions

You are arguing over definitions but it is useful. You make many posts that rely on these concepts so the definitions are relevant. That 'you are just arguing semantics' call is sometimes an irritating cached response.

but I seem to be at a loss to make people understand what I am trying to say here. You are, of course, welcome to tell me that this is my fault; but

You are making more than one claim here. The different-concept-alien stuff you have explained quite clearly (eg. from the first semicolon onwards in the parent). This seems to be obviously true. The part before the semicolon is a different concept (probably two). Your posts have not given me the impression that you consider the true<islam> issue distinct from normative issue and subjectivity. You also included 'objective morality' in with 'True', 'transcendental' and 'ultimate' as things that have no meaning. I believe you are confused and that your choice of definition for 'should' contributes to this.

it is somewhat disconcerting to find everyone saying that they agree with me

I say I disagree with a significant part of your position, although not the most important part.

, while continuing to disagree with each other.

I definitely disagree with Tim. I may agree with some of the others.

Comment author: wedrifid 19 November 2009 08:02:28PM 0 points [-]

The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.

It is the claims along the lines of 'truth value' that are most counterintuitive. The universality that you attribute to 'Right' also requires some translation.

Eliezer_Yudkowsky comments on A Less Wrong singularity article? - Less Wrong

Article Navigation

Comments (210)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (210)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login