Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Vladimir_Nesov comments on Can't Unbirth a Child - Less Wrong

24 Post author: Eliezer_Yudkowsky 28 December 2008 05:00PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (94)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Vladimir_Nesov 30 December 2008 06:29:25PM 0 points [-]

Implementing an algorithm is simpler than optimizing for morality: you have all kinds of equivalence at your disposal, you can undo anything. If the first AI doesn't itself contribute any moral content, you (or it) is free to renormalize it in any way, recreating it the way it was supposed to be built, as opposed to the way it was actually built, experimenting with its implementation, emulating its runs, and so on and so forth. If, on the other hand, its structure is morally significant, rebuilding might no longer be an option, and a final result may be worse than what it'd be possible to create starting from a morally blank slate (for the AI implementation). Morality is not time-reversible, and making a moral mistake at the point that is to guide the dynamic of moral growth for the future may be much more costly than it looks on the surface. Giving most of the universe for "paperclipping" with it being morally wrong to not give it away to the new mind is a real possibility, so we'd better avoid taking responsibility before understanding how reversible or irreversible the decision will turn out to be.