quartz comments on Q&A with experts on risks from AI #1 - Less Wrong

29 Post author: XiXiDu 08 January 2012 11:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (66)

You are viewing a single comment's thread.

Comment author: quartz 08 January 2012 04:10:16PM *  6 points [-]

my firmest belief about the timeline for human-level AI is that we can't estimate it usefully. partly this is because i don't think "human level AI" will prove to be a single thing (or event) that we can point to and say "aha there it is!". instead i think there will be a series of human level abilities that are achieved.

This sounds right. SIAI communications could probably be improved by acknowledging the incremental nature of AI development more explicitly. Have they addressed how this affects safety concerns?

Comment author: Vladimir_Nesov 08 January 2012 07:16:03PM *  5 points [-]

Science of how to make an AGI is developed gradually, with many prototypes along the way, but the important threshold is where it becomes possible to make a system that can continue open-ended development on its own (if left undisturbed and provided with moderate amount of computing resources). Some time after that point, it may become impossible to stop such a system, and if it ends up developing greater and greater advantage over time, without holding beneficial values, humanity eventually loses. It's the point where the process starts becoming more and more dangerous on its own, until it "explodes" in our faces, like supercritical mass of fissile material.