RobbBB comments on The genie knows, but doesn't care - Less Wrong

54 Post author: RobbBB 06 September 2013 06:42AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (515)

You are viewing a single comment's thread. Show more comments above.

Comment author: RobbBB 10 September 2013 04:51:24PM 0 points [-]

But it will scare friendly ones, which will want to keep their values stable.

Yes. If an AI is Friendly at one stage, then it is Friendly at every subsequent stage. This doesn't help make almost-Friendly AIs become genuinely Friendly, though.

It takes stupidity to misinterpret friendlienss.

Yes, but that's stupidity on the part of the human programmer, and/or on the part of the seed AI if we ask it for advice. The superintelligence didn't write its own utility function; the superintelligence may well understand Friendliness perfectly, but that doesn't matter if it hasn't been programmed to rewrite its source code to reflect its best understanding of 'Friendliness'. The seed is not the superintelligence. See: http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/