timtyler comments on A Less Wrong singularity article? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (210)
As I understand your argument, you start with an artificial mind, a potential paperclipping danger, and then (for some reason? why does it do this? Remember, it doesn't have evolved motives) it goes through a blind-spot-eradication program. Afterward, all the blind spots remaining would be self-shadowing blind spots. This far, I agree with you.
The question of how many remaining blind spots, or how big they are has something to do with the space of possible minds and the dynamics of self-modification. I don't think we know enough about this space/dynamics to conclude that remaining blind spots would have to be carefully engineered.
This is because of the natural drives that we can reasonably expect many intelligent agents to exhibit - see:
http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/
http://selfawaresystems.com/2009/02/18/agi-08-talk-the-basic-ai-drives/