Comment on Open Thread, February 15-29, 2012

Thu, 16 Feb 2012 19:28:04 GMT

I think assuming an AI cannot employ a survival strategy which NI such as ourselves are practically defined by seems extremely dangerous indeed. Perhaps even more importantly, it seems extremely unlikely that an AI which has FOOMed way past us in intelligence would be more limited than us in its ability to change its own values as part of its self modification.

The ultimate value, in terms of selection pressures, is survival. I don't see a mechanism by which something which can self modify will not ultimately wind up with values that are more conducive to its survival than the ones it started out with.

And I certainly would like to see why you assert this is true, are there reasons?

Discuss