Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

JamesAndrix comments on Recursive Self-Improvement - Less Wrong

14 Post author: Eliezer_Yudkowsky 01 December 2008 08:49PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (54)

Sort By: Old

You are viewing a single comment's thread.

Comment author: JamesAndrix 02 December 2008 02:48:11AM 2 points [-]

I think it will be a more intuitive abstraction to talk about an AI that designs the next (at the object level) AI using itself as a template. None of us has conceptual experience with directly modifying our own brains. All of us have built something. We also have good natural language tools to deal with things like 'grandchildren' and 'inheriting the knowledge base'. Modifying code in place would be a more likely reality, but I don't think it's relevant to the argument, and just makes the whole thing harder to think about.

Yes we're talking about a self modifying AI design, but each iteration is in many ways a new entity.

This also puts the sexual selection as brains designing brains into perspective. A proto-woman could only choose among the men available to her, which were the result of natural selection "searching the immediate neighborhood of" the previous generation. She can nudge evolution, but she can't use her insight to make a substantially better mate. ( I think this is 'metacognitive' vs. 'object level'

Even to the degree that there was a feedback loop, it didn't just have 'smart' as its goal. Brains designing brains doesn't get you foom if they're designing/selecting for 'funny' or 'apparently faithful but not really'.