Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Andrew_Ducker comments on Recursive Self-Improvement - Less Wrong

14 Post author: Eliezer_Yudkowsky 01 December 2008 08:49PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (54)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Andrew_Ducker 01 December 2008 10:16:52PM 2 points [-]

The problem, as I see it, is that you can't take bits out of a running piece of software and replace them with other bits, and have them still work, unless said piece of software is trivial.

The likelihood that you could change the object retrieval mechanism of your AI and have it still be the "same" AI, or even a reasonably functional one, is very low, unless the entire system was deliberately written in an incredibly modular way. And incredibly modular systems are _not_ efficient, which makes it unlikely that any early AI will be written in that manner.

The human brain is a mass of interconnecting systems, all tied together in a mish-mash of complexity. You couldn't upgrade any one part of it by finding a faster replacement for any one section of it. Attempting to perform brain surgery on yourself is going to be a slow, painstaking process, leaving you with far more dead AIs than live ones.

And, of course, as the AI picks the low-fruit of improvements, it'll start running into harder problems to solve that it may well find itself needing _more_ effort and attempts to solve.

Which doesn't mean it isn't possible - it just means that it's going to be a slow takeoff, not a fast one.

Comment author: gwern 14 October 2010 04:20:29PM 3 points [-]

And incredibly modular systems are not efficient, which makes it unlikely that any early AI will be written in that manner.

Whole-program compilation is all about collapsing modularity into an efficient spaghetti mess, once modularity has served its purpose with information-hiding and static checks.