gwern comments on AI prediction case study 5: Omohundro's AI drives - Less Wrong

5 Post author: Stuart_Armstrong 15 March 2013 09:09AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (5)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 18 March 2013 03:42:10PM 0 points [-]

trading cognition for pleasure seems like a reasonable example of not valuing, or failing to act on the value of, being more rational.

I think it's the same thing as before. AI drives is about a particular set of behaviors being an instrumental value for a large subset of all plausible agents; rationality is one of these instrumental (and not terminal) drives.

Providing an instance where an agent trades off an instrumental good (rationality) for a terminal good (pleasure) is simply not a counter-example - what else would an agent do when offered such a tradeoff? It would be like saying "supposedly, people earn money so as to spend it on things they want; but look! they're spending money on things like trips to Tahiti! Clearly that is not why they really earn money..."