You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

SilentCal comments on Is there a recursive self-improvement hierarchy? - Less Wrong Discussion

7 Post author: PhilGoetz 29 October 2015 02:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (10)

You are viewing a single comment's thread.

Comment author: SilentCal 30 October 2015 04:00:09PM 1 point [-]

At the heart of this question is some concept of resource permission that I'm trying to nail down--that is, agent X has 'self-modified' into agent Y iff agent Y has the same hardware resources that agent X had. This distinguishes self-modification from emulation, which is important; humans have limited self-modification, but with a long paper tape we can emulate any program.

A proposed measure: Define the 'emulation penalty' of a program that could execute on the AI's machine as the ratio of the runtime of the AI's fastest possible emulation of that program to the runtime of the program executing directly on the machine. The maximum emulation penalty over all possible programs puts at least an lower bound on the AI's ability to effectively self-modify into any possible agent.

An AI that can write and exec assembly would have a max emulation penalty of 1; one that can write and exec a higher-level language would probably have 10-100 (I think?); and one that could only carry out general computation by using an external paper tape would have a max emulation penalty in the billions or higher.

Comment author: Gurkenglas 09 January 2016 11:38:52AM 0 points [-]

Therefore, for a computer in Greg Egan's Permutation City, emulation is self-modification?