You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Risto_Saarelma comments on Charles Stross: Three arguments against the singularity - Less Wrong Discussion

10 Post author: ciphergoth 22 June 2011 09:52AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (54)

You are viewing a single comment's thread.

Comment author: Risto_Saarelma 22 June 2011 11:28:36AM *  3 points [-]

He seems to be pretty optimistic about an artificial intelligence architecture requiring lots of custom-crafted crunchy bits to go from being able to autonomously clean and fix your apartment, buy you groceries, make you lunch, drive you around town and figure out it needs to save your life and do so when you get a cardiac arrest to being able to do significant runaway learning and reasoning that will have serious and cumulative unintended consequences. And per Omohundro's Basic AI Drives paper, needing to have efficient real-world intelligence might count as a potentially dangerous motivation on its own, so not giving the things a goal system cribbed from humans won't help much.