endoself comments on A taxonomy of Oracle AIs - Less Wrong

13 Post author: lukeprog 08 March 2012 11:14PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (51)

You are viewing a single comment's thread. Show more comments above.

Comment author: endoself 09 March 2012 05:06:56AM *  -2 points [-]

would a true Oracle AI that didn't understand that humans could get bored really be that bad?

The important difficulty is not in making an AGI realize that humans can get bored, it is in making it realize that this is undesirable. Also, it would have to be able to figure out whether it is desirable to remove humans' boredom, effectively wireheading them in order to make them willing to enjoy repetitive activities.

I do agree that this could be less of a problem for an Oracle non-AI than for a goal-directed system, depending on other details of the design.