Eliezer_Yudkowsky comments on Inferring Values from Imperfect Optimizers - Less Wrong

2 Post author: nigerweiss 29 December 2012 10:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (20)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 31 December 2012 11:54:29AM 4 points [-]

given the choice between a large cost to optimize whatever you care about, or small cost to just optimize its own sense experiences, will prefer the latter.

You built the machine to optimize its sense experiences. It is not constructed to optimize anything else. That is just what it does. Not when it's cheaper, not when it's inconvenient to do otherwise, but at all times universally.