timtyler comments on AI indifference through utility manipulation - Less Wrong

4 Post author: Stuart_Armstrong 02 September 2010 05:06PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (53)

You are viewing a single comment's thread. Show more comments above.

Comment author: timtyler 06 September 2010 04:44:33PM *  0 points [-]

The utility function is always the same in this kind of scenario - and is not "updated".

It typically says something roughly like: stop button not pressed: business as normal - stop button pressed: let the engineeres dismantle your brain. That doesn't really let you be money-pumped because - for one thing, a pump needs repeated cycles to do much work. Also, after being switched off the agent can't engage in any economic activities.

Agent's won't get rid of such stipulations as they self-improve - under the assumption that a self-improving agent successfully preserves its utility function. Changing the agent's utility function would typically be very bad - from the point of view of the agent.