George_Weinberg comments on The Truly Iterated Prisoner's Dilemma - Less Wrong

18 Post author: Eliezer_Yudkowsky 04 September 2008 06:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (85)

Sort By: Old

You are viewing a single comment's thread.

Comment author: George_Weinberg 05 September 2008 09:10:00PM 0 points [-]

Change the problem and you change the solution.

If we assume that Eli and Clippy are both essentially self-modifying programs capable of verifiably publishing their own source codes, then indeed they can cooperate:

Eli modifies his own source code in such a way that he assures Clippy that his cooperation is contingent on Clippy's revealing his own source code and that the source code fulfills certain criteria, Clippy modifies his source code appropriately and publishes it.

Now each knows the other will cooperate.

But I think that although we in some ways resemble self-modifying computers, we cannot arbitrarily modify our own source codes, nor verifiably publish them. It's not at all clear to me that it would be a good thing if we could. Eliezer has constructed a scenario in which it would be favorable to be able to do so, but I don't think it would be difficult to construct a scenario in which it would be preferable to lack this ability.