TheOtherDave comments on Superintelligent AGI in a box - a question. - Less Wrong

14 Post author: Dmytry 23 February 2012 06:48PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (77)

You are viewing a single comment's thread. Show more comments above.

Comment author: TheOtherDave 25 February 2012 02:28:44AM 1 point [-]

If I accept the premise that it is programmed in such a way that it reports its internal processes completely and honestly, then I agree it can't "hide" its thoughts.

That said, if we're talking about a superhuman intelligence -- or even a human-level intelligence, come to that -- I'm not confident that we can reliably predict the consequences of its thoughts being implemented, even if we have detailed printouts of all of its thoughts and were willing to scan all of those thoughts looking for undesirable consequences of implementation before implementing them.