Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

MichaelVassar comments on Only humans can have human values - Less Wrong

34 Post author: PhilGoetz 26 April 2010 06:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (159)

You are viewing a single comment's thread. Show more comments above.

Comment author: MichaelVassar 27 April 2010 05:01:20AM 8 points [-]

This is extremely confused. Wireheading is an evolutionary dead-end because wireheads ignore their surroundings. Paperclippers, and for that matter, staplers and FAIs pay exclusive attention to their surroundings and ignore their terminal utility functions except to protect them physically. It's just that after acquiring all the resources available, clippy makes clips and Friendly makes things that humans would want if they thought more clearly, such as the experience of less clear thinking humans eating ice cream.

Comment author: PhilGoetz 27 April 2010 10:03:41PM *  -3 points [-]

such as the experience of less clear thinking humans eating ice cream

If the goal is to give people the same experience that they would get from giving ice cream, is it satisfied by giving them a button they can press to get that experience?

Comment author: MichaelVassar 01 May 2010 02:19:49AM 1 point [-]


Comment author: PhilGoetz 01 May 2010 02:58:25AM 0 points [-]

I would call that wireheading.

Comment author: NancyLebovitz 01 May 2010 09:03:47AM 1 point [-]

It's only wireheading if it becomes a primary value. If it's just fun subordinate to other values, it isn't different from "in the body" fun.

Comment author: PhilGoetz 01 May 2010 03:22:50PM -1 points [-]

What's a primary value? This sounds like a binary distinction, and I'm always skeptical of binary distinctions.

You could say the badness of the action is proportional to the fraction of your time that you spend doing it. But for that to work, you would assign the action the same bad value per unit time.

Are you saying that wireheading and other forms of fun are no different; and all fun should be pursued in moderation? So spending 1 hour pushing your button is comparable to spending 1 hour attending a concert?

Comment author: PhilGoetz 27 April 2010 04:32:55PM *  -3 points [-]

(That's only a paperclipper with no discounting of the future, BTW.)

Paperclippers are not evolutionarily viable, nor is there any plausible evolutionary explanation for paperclippers to emerge.

You can posit a single artificial entity becoming a paperclipper via bad design. In the present context, which is of many agents trying to agree on ethics, this single entity has only a small voice.

It's legit to talk about paperclippers in the context of the danger they pose if they become a singleton. It's not legit to bring them up outside that context as a bogeyman to dismiss the idea of agreement on values.