Lumifer comments on Open thread, Aug. 10 - Aug. 16, 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (283)
Well, think about it. We are talking about a self-improving AI. It literally changes itself. You start with a seed AI, let's call it AI-0, and it bootstraps itself to an omnipotent AI which we can call AI-1.
Note that the programmers have no idea how to construct AI-1. They have no idea about the path from AI-0 to AI-1. All they (and we) know is that AI-0 and AI-1 will very very different.
Given this, I don't think that the program will be a map to the future. I don't think that the concept of "retaining admin" would even make sense for an AI-1. It will be completely different from what it started as. And I fail to see why you have a firm belief that it will be docile and obedient.