RobinZ comments on A Less Wrong singularity article? - Less Wrong

28 Post author: Kaj_Sotala 17 November 2009 02:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (210)

Tags: singularity ai crowdsourcing

Comments (210)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 18 November 2009 02:24:24AM 5 points [-]

Correct. I'm a moral cognitivist; "should" statements have truth-conditions. It's just that very few possible minds care whether should-statements are true or not; most possible minds care about whether alien statements (like "leads-to-maximum-paperclips") are true or not. They would agree with us on what should be done; they just wouldn't care, because they aren't built to do what they should. They would similarly agree with us that their morals are pointless, but would be concerned with whether their morals are justified-by-paperclip-production, not whether their morals are pointless. And under ordinary circumstances, of course, they would never formulate - let alone bother to compute - the function we name "should" (or the closely related functions "justifiable" or "arbitrary").

Comment author: RobinZ 18 November 2009 10:12:30PM * 4 points [-]

Am I right in interpreting the bulk of the thread following this comment (excepting perhaps the FAI derail) as a dispute on the definition of "should"?

Comment author: CronoDAS 19 November 2009 02:58:05AM 2 points [-]

Yes, we're disputing definitions, which tends to become pointless. However, we can't seem to get away from using the word "should", so we might as well get it pinned down to something we can agree upon.

Comment author: wedrifid 18 November 2009 10:52:52PM * 0 points [-]

Am I right in interpreting the bulk of the thread following this comment (excepting perhaps the FAI derail) as a dispute on the definition of "should"?

I think you are right.

The dispute also serves as a signal of what some parts of the disputants personal morality probably includes. This is fitting with the practical purpose that the concept 'should' has in general. Given what Eliezer has chosen as his mission this kind of signalling is a matter of life and death in the same way that it would have been in our environment of evolutionary adaptation. That is, if people had sufficient objection to Eliezer's values they would kill him rather than let him complete an AI.

Comment author: timtyler comment score below threshold [+] (37 children)

Comment author: Eliezer_Yudkowsky 18 November 2009 11:34:48PM 3 points [-]

As you can see, RobinZ, I'm trying to cure a particular kind of confusion here. The way people deploy their mental categories has consequences. The problem here is that "should" is already bound fairly tightly to certain concepts, no matter what sort of verbal definitions people think they're deploying, and if they expand the verbal label beyond that, it has consequences for e.g. how they think aliens and AIs will work, and consequences for how they emotionally experience their own moralities.

Comment author: timtyler 19 November 2009 07:19:51PM * 5 points [-]

It is odd how you apparently seem to think you are using the conventional definition of "should" - when you have several people telling you that your use of "should" and "ought" is counter-intuitive.

Most people are familiar with the idea that there are different human cultures, with somewhat different notions of right and wrong - and that "should" is often used in the context of the local moral climate.

For example:

If the owner of the restaurant serves you himself, you should still tip him;
You should not put your elbows on the table while you are eating;
Women should curtsey - "a little bob is quite sufficient".

Comment author: wedrifid 19 November 2009 07:47:04PM * 3 points [-]

It is odd how you apparently seem to think you are using the conventional definition of "should" - when you have several people telling you that your use of "should" and "ought" is counter-intuitive.

To be fair, there are several quite distinct ways in which 'should' is typically used. Eliezer's usage is one of them. It is used more or less universally by children and tends to be supplanted or supplemented as people mature with the 'local context' definition you mention and/or the 'best action for agent given his preferences' definition. In Eliezer's case he seems to have instead evolved and philosophically refined the child version. (I hasten to add that I imply only that he matured his moral outlook in other ways than by transitioning usage of those particular words in the most common manner.)

Comment author: timtyler 19 November 2009 07:58:53PM 4 points [-]

I can understand such usage. However, we have things like: "I'm trying to cure a particular kind of confusion here". The confusion he is apparently talking about is the conventional view of "ought" and "should" - and it doesn't need "curing".

In fact, it helps us to understand the moral customs of other cultures - rather than labeling them as being full of "bad" heathens - who need to be brought into the light.

Comment author: Eliezer_Yudkowsky 19 November 2009 07:26:30PM * 0 points [-]

My use is not counterintuitive. The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.

If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.

Comment author: wedrifid 19 November 2009 07:51:38PM * 5 points [-]

If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.

One of the standard usages is "doing this will most enhance your utility". As in "you should kill that motherf@#$%". This is distinct from 'right' and 'good' although 'ought' is used in the same way, albeit less frequently. It is advice, rather than exhortation.

Comment author: CronoDAS 19 November 2009 07:57:28PM * 7 points [-]

Indeed. "The Pebblesorters should avoid making piles of 1,001 stones" makes perfect sense.

Comment author: timtyler 19 November 2009 08:06:35PM 0 points [-]

"Should" and "ought" actually have strong connotations of societal morality.

Should you rob the bank? Should you have sex with the minor? Should you confess to the crime?

Your personal utility is one thing - but "should" and "ought" often have more to do with what society thinks of your actions.

Comment author: wedrifid 19 November 2009 08:25:24PM 0 points [-]

Should you rob the bank?

Probably not.

Should you have sex with the minor?

Probably not here.

Should you confess to the crime?

Hell no. "The Fifth" is the only significant law-item that I'm explicitly familiar with. And I'm not even American.

Your personal utility is one thing - but "should" and "ought" often have more to do with what society thinks of your actions.

More often what you want society to think of people's actions (either as a signal or as persuasion. I wonder which category my answers above fit into?).

Comment author: timtyler 19 November 2009 07:48:24PM * 5 points [-]

It's counterintuitive to me - and I'm not the only one - if you look at the other comments here.

Aliens could have the "right", "good", "ought" and "should" concept cluster - just as some other social animals can, or other tribes, or humans at other times.

Basically, there are a whole bunch of possible and actual moral frameworks - and these words normally operate relative to the framework under consideration.

There are some people who think that "right" and "wrong" have some kind of universal moral meaning. However most of those people are religious, and think morality comes straight from god - or some such nonsense.

Comment author: Eliezer_Yudkowsky comment score below threshold [+] (19 children)

Comment author: wedrifid 19 November 2009 08:02:28PM 0 points [-]

The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.

It is the claims along the lines of 'truth value' that are most counterintuitive. The universality that you attribute to 'Right' also requires some translation.

Comment author: RobinZ 19 November 2009 12:57:06AM 2 points [-]

The problem here is that "should" is already bound fairly tightly to certain concepts, no matter what sort of verbal definitions people think they're deploying, and if they expand the verbal label beyond that, it has consequences for e.g. how they think aliens and AIs will work, and consequences for how they emotionally experience their own moralities.

I see, and that is an excellent point. Daniel Dennett has taken a similar attitude towards qualia, if I interpret you correctly - he argues that the idea of qualia is so inextricably bound with its standard properties (his list goes ineffable, intrinsic, private, and directly or immediately apprehensible by the consciousness) that to describe a phenomenon lacking those properties by that term is as wrongheaded as using the term elan vital to refer to DNA.

I withdraw my implied criticism.

Comment author: wedrifid 18 November 2009 11:19:44PM * 1 point [-]

An intelligent machine might make one of its first acts the assassination of other machine intelligence researchers - unless it is explicitly told not to do that. I figure we are going to want machines that will obey the law. That should be part of any sensible machine morality proposal.

I absolutely do not want my FAI to be constrained by the law. If the FAI allows machine intelligence researchers to create an uFAI we will all die. An AI that values the law above the existence of me and my species is evil, not Friendly. I wouldn't want the FAI to kill such researchers unless it was unable to find a more appealing way to ensure future safety but I wouldn't dream of constraining it to either laws or politics. But come to think of it I don't want it to be sensible either.

The Three Laws of Robotics may be a naive conception but that Zeroth law was a step in the right direction.

Comment author: timtyler 19 November 2009 06:40:15PM 0 points [-]

Re: If the FAI allows machine intelligence researchers to create an uFAI we will all die

Yes, that's probably just the kind of paranoid delusional thinking that a psychopathic superintelligence with no respect for the law would use to justify its murder of academic researchers.

Hopefully, we won't let it get that far. Constructing an autonomous tool that will kill people is conspiracy to murder - so hopefully the legal system will allow us to lock up researchers who lack respect for the law before they do some real damage. Assassinating your competitors is not an acceptable business practice.

Hopefully, the researchers will learn the error of their ways before then. The first big and successful machine intelligence project may well be a collaboration. Help build my tool, or be killed by it - is a rather aggressive proposition - and I expect most researchers will reject it, and expend their energies elsewhere - hopefully on more law-abiding projects.

Comment author: wedrifid 19 November 2009 07:21:32PM * 4 points [-]

Yes, that's probably just the kind of paranoid delusional thinking that a psychopathic superintelligence with no respect for the law would use to justify its murder of academic researchers.

You seem confused (or, perhaps, hysterical). A psychopathic superintelligence would have no need to justify anything it does to anyone.

By including 'delusional' you appear to be claiming that an unfriendly super-intelligence would not likely cause the extinction of humanity. Was that your intent? If so, why do you suggest that the first actions of a FAI would be to kill AI researchers? Do you believe that a superintelligence will disagree with you about whether uFAI is a threat and that it will be wrong while you are right? That is a bizarre prediction.

and I expect most researchers will reject it, and expend their energies elsewhere - hopefully on more law-abiding projects.

You seem to have a lot of faith in the law. I find this odd. Has it escaped your notice that a GAI is not constrained by country borders? I'm afraid most of the universe, even most of the planet, is out of your jurisdiction.

Comment author: timtyler 24 November 2009 10:26:25PM * -2 points [-]

Re: You seem confused (or, perhaps, hysterical).

Uh, thanks :-(

A powerful corporate agent not bound by the law might well choose to assassinate its potential competitors - if it thought it could get away with it. Its competitors are likely to be among those best placed to prevent it from meeting its goals.

Its competitors don't have to want to destroy all humankind for it to want to eliminate them! The tiniest divergence between its goals and theirs could potentially be enough.

Comment author: MichaelVassar 19 November 2009 04:34:58PM 0 points [-]

It is a misconception to think of law as a set of rules. Even more so to understand them as a set of rules that apply to non-humans today. In addition, rules won't be very effective constraints on superintelligences.

RobinZ comments on A Less Wrong singularity article? - Less Wrong

Article Navigation

Comments (210)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (210)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login