Comment on Evaluating the feasibility of SI's plan

Thu, 10 Jan 2013 13:49:44 GMT

Thanks for your comment. But what does this tell us about SI's current R&D strategy and whether they should modify it?

a non-god AI is safer still, and I think that's the kind of AI most research is going into. .... AIs which aren't programmed with a desire to self-modify

A nascent AGI will self-improve to godlike levels. This is true even if it is not programmed with a desire to self-modify, since self-improvement is a useful technique in achieving other goals.

In general, AIs that can't self-modify are going to be safer than those which can

An interesting approach -- design an AI so that it can't self-modify. I don't know that I've seen that approach designed in detail. Seems worth at least an article.

non-god AI ... the kind of AI most research...

Good point. Most AGI developers (there are not too many) are not seriously considering the post-Explosion stage. Even the few who are well aware of the possibility don't treat it seriously in their implementation work. But that doesn't mean that (if and when they succeed in making a nascent AGI) it won't explode.

Discuss