kilobug comments on Rationality Quotes September 2012 - Less Wrong

7 Post author: Jayson_Virissimo 03 September 2012 05:18AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (1088)

You are viewing a single comment's thread. Show more comments above.

Comment author: kilobug 07 September 2012 02:41:12PM -1 points [-]

If you consider three class of genies :

  • (A) a genie that's going to be "just be a dick" but is not skilled at it ;

  • (B) a genie that is benevolent ;

  • (C) a genie that's going to be "just be a dick" but is very skilled at it.

Your test will (may at least) tell apart A from (B or C). It won't tell apart B from C.

The "there is no safe wish" rule applies to C. Sure, if your genie is not skilled a being "evil" (having an utility function very different from yours), you can craft a wish that is beyond the genie's ability to twist it. But if the genie is skilled, much more intelligent than you are, with like the ability to spend the equivalent of one million of years of thinking how to twist the wish in one second, he'll find a flaw and use it.