atucker comments on The Friendly AI Game - Less Wrong

38 Post author: bentarm 15 March 2011 04:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (170)

You are viewing a single comment's thread. Show more comments above.

Comment author: atucker 16 March 2011 01:41:56AM *  1 point [-]

Assuming that the utility function is written in a way that makes loss of utility possible (utility = dollars in bank or something), this is a failure mode:

AI stops short of the limit, makes another AI that prevents loss of utility, hits the bound, and then shuts down.

Second AI takes over the universe as a precaution against any future disutility.