Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

ShardPhoenix comments on Devil's Offers - Less Wrong

21 Post author: Eliezer_Yudkowsky 25 December 2008 05:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (46)

Sort By: Old

You are viewing a single comment's thread.

Comment author: ShardPhoenix 25 December 2008 04:56:13PM 0 points [-]

@James:

Doesn't the choice of a perfect external regulator amount to the same thing as directly imposing restrictions on yourself, thereby going back to the original problem? I suppose such a regulator, on indeed any stabilizing self-modification, could have the advantage of being publicly available and widely used, and therefore be well-tested and having thoroughly understood operations and consequences.

Another way to do it might be to create many copies of yourself (I'm assuming this scenario takes place inside a computer) and let majority (or 2/3s majority or etc) rule when it comes to "rescuing" copies that have made un-self-recoverable errors.

Anyway I suppose this is all somewhat beside the point since such a scenario was chosen as an example of what Eliezer expects a successful future to *not* look like.