royf comments on Fake Causality - Less Wrong

41 Post author: Eliezer_Yudkowsky 23 August 2007 06:12PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (86)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: royf 05 June 2012 02:56:43AM *  0 points [-]

I don't think "surviving" is a well-defined term here. Every time you self-modify, you replace yourself with a different agent, so in that sense any agent that keeps surviving is one that does not self-modify.

I placed "survive" in quotation marks to signal that I was aware of that, and that I meant "the other thing". I didn't realize that this was far from clear enough, sorry.

For lack of better shared terminology, what I meant by "surviving" is continuing to be executable. Self modification is not suicide, you and I are doing it all the time.

Can I write a program that begins by computing the cluster of all agents similar to it, and switches to the next one (lexicographically) every 24 hours?

No, you cannot. This function is non-computable in the Turing sense.

A computable limited version of it (whatever it is) could be possible. But this particular agent cannot modify itself "in any way it wants", so it's consistent with my proposition.

The natural objection is that there is one part of the agent's state that is inviolate in this example: the 24-hour rotation period

This is a very weak limitation of the space of possible modifications. I meant a much stronger one.

But I'm skeptical that this limitation can be encoded mathematically.

This weak limitation is easy to formalize.

The stronger limitation I'm thinking of is challenging to formalize, but I'm pretty confident that it can be done.

Comment author: Kindly 05 June 2012 03:22:32AM 0 points [-]

No, you cannot. This function is non-computable in the Turing sense.

Aha! I think this is the important bit. I'll have to think about this, but it's probably what the problem is.