Technologos comments on Reason as memetic immune disorder - Less Wrong

215 Post author: PhilGoetz 19 September 2009 09:05PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (166)

You are viewing a single comment's thread. Show more comments above.

Comment author: Technologos 21 September 2009 07:31:34AM 3 points [-]

And not just by persuading the guards--the kind of AIs we are talking about, transhuman-level AIs, could potentially do all kinds of mind-hacking things of which we haven't even yet conceived. Hell, they could do things that we will never be able to conceive unaided.

If we ever set up a system that relies on humans restraining a self-modifying AI, we had better be sure beforehand that the AI is Friendly. The only restraints that I can think of that would provably work involve limiting the AIs access to resources so that it never achieves a level of intelligence equal to or higher than human--but then, we haven't quite made an AI, have we? Not much benefit to a glorified expert system.

If you haven't read the AI Box experiment reports I linked above, I recommend them--apparently, it doesn't quite take a transhuman-level AI to get out of a "test harness."