RobinZ comments on You cannot be mistaken about (not) wanting to wirehead - Less Wrong

34 Post author: Kaj_Sotala 26 January 2010 12:06PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (79)

You are viewing a single comment's thread. Show more comments above.

Comment author: LauraABJ 26 January 2010 04:44:10PM 5 points [-]

Your examples of getting tired after sex or satisfied after eating are based on current human physiology and neurochemistry, which I think most people here are assuming will no longer confine our drives after AI/uploading. How can you be sure what you would do if you didn't get tired?

I also disagree with the idea that 'pleasure' is what is central to 'wireheading.' (I acknowledge that I may need a new term.) I take the broader view that wireheading is getting stuck in a positive feed-back loop that excludes all other activity, and for this to occur, anything positively-reinforcing will do.* For example, let's say Jane Doe wants to want to exercise, and so modifies her preferences. Now lets say this modification is not calibrated correctly, and so she ends up on the treadmill 24/7, never wanting to get off of it. Though the activity is not pleasurable, she is still stuck in the loop. Even if we would not make a mistake quite this mundane, it is not difficult to imagine similar problems occurring after a few rounds of 'preference modification' by free transhumans. If someone has a drive to be satisfied, then satisfied he shall be, one way or another. Simple solutions, like putting in a preference for complexity, may not be sufficient safeguards either. Imagine an entity that spends all of its time computing and tracing infinite fractiles. Pinnacle of human evolution or wirehead?

*Disclaimer: I haven't yet defined the time parameters. For example, if the loop takes 24 hours to complete as opposed to a few seconds, is it still wireheading? What about 100 years? But I think the general idea is important to consider.

Comment author: RobinZ 26 January 2010 05:33:14PM *  2 points [-]

Your examples of getting tired after sex or satisfied after eating are based on current human physiology and neurochemistry, which I think most people here are assuming will no longer confine our drives after AI/uploading. How can you be sure what you would do if you didn't get tired?

I got bored with playing Gran Turismo all the time in less than a week - the timescale might change, but eventually blessed boredom would rescue me from such a loop.

Edit: From most known loops of this type - I agree with your concern about loops in general.