You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Vladimir_Nesov comments on FAI FAQ draft: What is Friendly AI? - Less Wrong Discussion

2 Post author: lukeprog 13 November 2011 07:26PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (11)

You are viewing a single comment's thread.

Comment author: Vladimir_Nesov 14 November 2011 01:30:32PM *  2 points [-]

The "features possessed by any superintelligence" paragraphs seem particularly vulnerable to antagonistic reading:

Superpower: a superintelligent machine will have unprecedented powers to reshape reality, and therefore will achieve its goals with highly efficient methods that confound human expectations and desires.

Superintelligent machines have mystical powers allowing them to transcend laws of physics and reality.

It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety

It will refuse to do anything unless rules are precise. It will ignore details of designers' specifications out of spite.