Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Nathan_Kurz comments on Surprised by Brains - Less Wrong

22 Post author: Eliezer_Yudkowsky 23 November 2008 07:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (25)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Nathan_Kurz 23 November 2008 05:11:43PM 0 points [-]

Eliezer ---

Do you think this analogy is useful for estimating the value of friendliness? That is, is the impact of humans on other species and our environment during this explosion of intelligence a useful frame for pondering the impact of a rapidly evolving AI on humans?

I think it has potential to be useful, but I'm not sure in which direction it should be read.

While we've driven some species toward extinction, others have flourished. And while I'm optimistic that as intelligence increases we'll be better able to control our negative impacts on the environment, I'm also worried that as the scale of our impacts increases a single mistake could be fatal.