Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Matt_Simpson comments on Only humans can have human values - Less Wrong

33 Post author: PhilGoetz 26 April 2010 06:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (159)

You are viewing a single comment's thread.

Comment author: Matt_Simpson 27 April 2010 02:09:04PM *  4 points [-]

First of all, good post.

My main response is, aren't we running on hostile hardware? I am not the entire system called "Matt Simpson." I am a program, or a particular set of programs, running on the system called "Matt Simpson." This system runs lots of other programs. Some, like automatically pulling my hand away after I touch a hot stove, happen to achieve my values. Others, like running from or attacking anything that vaguely resembles a snake, are a minor annoyance. Still others, like the system wanting to violently attack other males, work directly counter to my values.

The hard problem of ethics - determining what our values actually are - is (partially?) reduced to determining which parts of the system are me. Which is still a hard problem. (Despite my language, I don't really think this is an identity problem)

Comment author: PhilGoetz 27 April 2010 03:52:22PM 5 points [-]

I model this as a case where the Matt Simpson system has a network of preference-systems (where a "preference system" = propositions + algorithms + environment), and some of those preference systems are usually in the minority in the action recommendations they give. Matt Simpson the reflective agent would have less stress if he could eliminate those preference systems in the "loyal opposition".

Then, replace each preference system with a system that puts as much of its content as possible into propositions, so you can optimize algorithms.

You might find, after doing this, that those preferences in the "loyal opposition" were of great value in a small number of situations. The values we're ashamed of might be the "special teams" (American football term) of values, that are needed only rarely (yet vote all the time). I'm just speculating.

If that's still what you want to do, it's not honest to call the values left over "human values". Human values are the values humans have, in all their meaty messiness.

And you're still faced with the problem, for the preference systems that remain, of deciding what results they should give in new environments.

Comment author: Matt_Simpson 27 April 2010 06:38:50PM 1 point [-]

If that's still what you want to do, it's not honest to call the values left over "human values". Human values are the values humans have, in all their meaty messiness.

Fair enough

And you're still faced with the problem, for the preference systems that remain, of deciding what results they should give in new environments.

To put it simply (and more generally) I would just say that I still don't fully know my preferences. In particular, even after figuring out which parts of the Matt Simpson system are me, my values are still underdetermined (this is the point of my "partially?" in the grandparent).

I'm not disagreeing with you at all, btw, just clarifying some terms.

With that out of the way, are you suggesting that there is no correct/best way to go from the underdetermined preferences to a consistent set of preferences? Or in your terms, to decide what results the remaining preference systems should give in new environments? (I think I know your answer to this - its just an understanding check).

Comment author: thomblake 27 April 2010 02:18:23PM 5 points [-]

Agreed, and a lot of the disputes in this realm come from people drawing identity in different ways. I submit that I am the entire system called "Thom Blake" as well as a bunch of other stuff like my smartphone and car.

Once you give up Platonic identity (or the immortal soul) you also get to have much fuzzier boundaries. A dualist who thinks "I am my mind" cannot imagine losing a part of oneself without "no longer being me", and thinks that cutting off one's hand has no impact on "who I really am". I think that cutting off my hand will have a great impact on me, in terms of my self and my identity, and cutting off my right hand less so than my left.