Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Johnicholas comments on Surprised by Brains - Less Wrong

22 Post author: Eliezer_Yudkowsky 23 November 2008 07:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (25)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Johnicholas 24 November 2008 05:33:00PM 0 points [-]

"why would innovations discovered by a single AI not spread soon to the others, and why would a non-friendly AI not use those innovations to trade, instead of war?"

One of Elizer's central (and I think, indisputable) claims is that a hand-made AI, after going recursive self-improvement, could be powerful in the real world while being WILDLY unpredictable in its actions. It doesn't have to be economically rational.

Given a paperclip-manufacturing AI, busily converting earth into grey goo into paperclips, there's no reason to believe we could communicate with it well enough to offer to help.