orthonormal comments on The scourge of perverse-mindedness - Less Wrong

95 Post author: simplicio 21 March 2010 07:08AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (249)

You are viewing a single comment's thread. Show more comments above.

Comment author: orthonormal 22 March 2010 03:16:11AM *  2 points [-]

What Robin is saying is, there's a difference between

  • "metrics that correlate well enough with what you really want that you can make them the subject of contracts with other human beings", and

  • "metrics that correlate well enough with what you really want that you can make them the subject of a transhuman intelligence's goals".

There are creative avenues of fulfilling the letter without fulfilling the spirit that would never occur to you but would almost certainly occur to a superintelligence, not because xe is malicious, but because they're the optimal way to achieve the explicit goal set for xer. Your optimism, your belief that you can easily specify a goal (in computer code, not even English words) which admits of no undesirable creative shortcuts, is grossly misplaced once you bring smarter-than-human agents into the discussion. You cannot patch this problem; it has to be rigorously solved, or your AI wrecks the world.