Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Eliezer_Yudkowsky comments on Devil's Offers - Less Wrong

21 Post author: Eliezer_Yudkowsky 25 December 2008 05:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (46)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 08 January 2009 04:33:47PM 2 points [-]

Cameron, I suppose that's a fair enough comment. I'm used to the way things work in AI, where the naive simply fail utterly and completely to accomplish anything whatsoever, rather than hurting anyone else or themselves, and you have to get pretty far to get beyond that to the realm of dangers.

Not to mention, I'm used to the idiom that the lack of any prior feedback or any second try is what makes something an adult problem - but that really is representative of the dangers faced by someone able to modify their own brain circuitry. If there's an AI that can say "No" but you're allowed to ignore the "No" then one mistake is fatal. The sort of people who think, "Gee, I'll just run myself with the modification for a while and see what happens 'cuz I can always go back" - they might ignore a "No" based on their concept of "testing", and then that would be the end of them.

You want to put the dangerous things behind a challenge/lock such that by the time you pass it you know how dangerous they really are. "Make an AI", unfortunately, may not be quite strong enough as a case of this, but "Make an AI without anyone helping you on a 100MHz computer with 1GB of RAM and 100GB of disk space" is probably strong enough.