RobbBB comments on The genie knows, but doesn't care - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (515)
Yes. If an AI is Friendly at one stage, then it is Friendly at every subsequent stage. This doesn't help make almost-Friendly AIs become genuinely Friendly, though.
Yes, but that's stupidity on the part of the human programmer, and/or on the part of the seed AI if we ask it for advice. The superintelligence didn't write its own utility function; the superintelligence may well understand Friendliness perfectly, but that doesn't matter if it hasn't been programmed to rewrite its source code to reflect its best understanding of 'Friendliness'. The seed is not the superintelligence. See: http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/