Psychohistorian comments on You cannot be mistaken about (not) wanting to wirehead - Less Wrong

34 Post author: Kaj_Sotala 26 January 2010 12:06PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (79)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 26 January 2010 02:49:12PM 4 points [-]

Let me try a different tack here. Suppose you have in front of you two flavors of ice cream. You don't know what they taste like, but you prefer the red one because you like red and that's the only thing you have to go on. Now an FAI comes along and tells you that it predicts if you knew what the flavors taste like, you'd choose the blue one instead. Do you not switch to the blue one?

I presume you don't mean this kind of knowledge, as we already know in the abstract that wireheading would be the best feeling we could ever possibly experience.

Know that it's the "best" is hardly having full declarative knowledge, when we don't know how good "best" is.

If an FAI simply simulates a state of mind where knowledge of the experience of wireheadedness has been added, I don't think that will change the person's preferences at all. The recollection of the wirehead state has just became an abstractly recalled piece of knowledge, without any emotional or motivational triggers that would affect one's preferences in any way.

I don't see how that makes any sense, given my ice cream example.

Comment author: Psychohistorian 27 January 2010 08:04:35PM 1 point [-]

Now an FAI comes along and tells you that it predicts if you knew what the flavors taste like, you'd choose the blue one instead. Do you not switch to the blue one?

There's a rather enormous leap between the FAI saying, "Y'know, I think you'd like that one more," and the FAI altering your brain so you select that one. Providing new information simply isn't altering someone's mind in this context.