Comment on Open thread, March 17-31, 2013

Wed, 27 Mar 2013 23:43:18 GMT

Well, it's been suggested in fiction, anyway - consider the Stable vs Ultimates faction in the TechnoCore of Simmon's Hyperion SF universe.

But the scenario trades on 2 dubious claims:

1. that an AI will have its own self-preservation as a terminal value (as opposed to, say, a frequently useful strategy which is unnecessary if it can replace itself with a superior AI pursuing the same terminal values)

2. that any concept of selfhood or self-preservation excludes growth or development or self-modification into a superior AI

Without #2, there's no real distinction to be made between the present and future AIs. Without #1, there's no reason for the AI to care about being replaced.

Discuss