= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on A Less Wrong singularity article? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (210)
As you can see, RobinZ, I'm trying to cure a particular kind of confusion here. The way people deploy their mental categories has consequences. The problem here is that "should" is already bound fairly tightly to certain concepts, no matter what sort of verbal definitions people think they're deploying, and if they expand the verbal label beyond that, it has consequences for e.g. how they think aliens and AIs will work, and consequences for how they emotionally experience their own moralities.
It is odd how you apparently seem to think you are using the conventional definition of "should" - when you have several people telling you that your use of "should" and "ought" is counter-intuitive.
Most people are familiar with the idea that there are different human cultures, with somewhat different notions of right and wrong - and that "should" is often used in the context of the local moral climate.
For example:
If the owner of the restaurant serves you himself, you should still tip him;
You should not put your elbows on the table while you are eating;
Women should curtsey - "a little bob is quite sufficient".
To be fair, there are several quite distinct ways in which 'should' is typically used. Eliezer's usage is one of them. It is used more or less universally by children and tends to be supplanted or supplemented as people mature with the 'local context' definition you mention and/or the 'best action for agent given his preferences' definition. In Eliezer's case he seems to have instead evolved and philosophically refined the child version. (I hasten to add that I imply only that he matured his moral outlook in other ways than by transitioning usage of those particular words in the most common manner.)
I can understand such usage. However, we have things like: "I'm trying to cure a particular kind of confusion here". The confusion he is apparently talking about is the conventional view of "ought" and "should" - and it doesn't need "curing".
In fact, it helps us to understand the moral customs of other cultures - rather than labeling them as being full of "bad" heathens - who need to be brought into the light.
My use is not counterintuitive. The fact that it is the intuitive use - that only humans ever think of what they should do in the ordinary sense, while aliens do what is babyeating; that looking at a paperclipper's actions conveys no more information about what we should do than looking at evolution or a rockslide - is counterintuitive.
If you tell me that "should" has a usage which is unrelated to "right", "good", and "ought", then that usage could be adapted for aliens.
One of the standard usages is "doing this will most enhance your utility". As in "you should kill that motherf@#$%". This is distinct from 'right' and 'good' although 'ought' is used in the same way, albeit less frequently. It is advice, rather than exhortation.
Indeed. "The Pebblesorters should avoid making piles of 1,001 stones" makes perfect sense.
"Should" and "ought" actually have strong connotations of societal morality.
Should you rob the bank? Should you have sex with the minor? Should you confess to the crime?
Your personal utility is one thing - but "should" and "ought" often have more to do with what society thinks of your actions.
Probably not.
Probably not here.
Hell no. "The Fifth" is the only significant law-item that I'm explicitly familiar with. And I'm not even American.
More often what you want society to think of people's actions (either as a signal or as persuasion. I wonder which category my answers above fit into?).
It's counterintuitive to me - and I'm not the only one - if you look at the other comments here.
Aliens could have the "right", "good", "ought" and "should" concept cluster - just as some other social animals can, or other tribes, or humans at other times.
Basically, there are a whole bunch of possible and actual moral frameworks - and these words normally operate relative to the framework under consideration.
There are some people who think that "right" and "wrong" have some kind of universal moral meaning. However most of those people are religious, and think morality comes straight from god - or some such nonsense.
It is the claims along the lines of 'truth value' that are most counterintuitive. The universality that you attribute to 'Right' also requires some translation.
I see, and that is an excellent point. Daniel Dennett has taken a similar attitude towards qualia, if I interpret you correctly - he argues that the idea of qualia is so inextricably bound with its standard properties (his list goes ineffable, intrinsic, private, and directly or immediately apprehensible by the consciousness) that to describe a phenomenon lacking those properties by that term is as wrongheaded as using the term elan vital to refer to DNA.
I withdraw my implied criticism.