TimS comments on Open Thread, February 15-29, 2012 - Less Wrong

3 Post author: OpenThreadGuy 15 February 2012 06:00AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (194)

You are viewing a single comment's thread. Show more comments above.

Comment author: TimS 16 February 2012 08:16:20PM *  0 points [-]

Humans have changed values to maximize other values (such as survival) throughout history. That's cultural assimilation in a nutshell. But some people choose to maximize values other than survival (e.g. every martyr ever). And that hasn't always been pointless - consider the value to the growth of Christianity created by the early Christian martyrs.

If an AI were faced with the possibility of self-modifying to reduce its adherence to value Y in order to maximize value X, then we would expect the AI to do so only when value X was "higher priority" than value Y. Otherwise, we would expect the AI to choose not to self-modify.