Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

simon2 comments on Mirrors and Paintings - Less Wrong

12 Post author: Eliezer_Yudkowsky 23 August 2008 12:29AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (41)

Sort By: Old

You are viewing a single comment's thread.

Comment author: simon2 23 August 2008 06:31:01AM 1 point [-]

As previously mentioned, there are tricky aspects to this. You can't say: "You see those humans over there? Whatever desire is represented in their brains, is therefore right." This, from a moral perspective, is wrong - wanting something doesn't make it right - and the conjugate failure of the AI is that it will reprogram your brains to want things that are easily obtained in great quantity. If the humans are PA, then we want the AI to be PA+1, not Self-PA... metaphorically speaking.

Before reading this post, if I had been programming a friendly AI I would have attempted to solve this issue by programming the AI it to take into account only minds existing at the moment it makes its decisions. (the AI still cares about the future, but only to the extent that currently existing minds, if extrapolated, would care about the future). This technique has the flaw that it would be likely to fail in the event that time travel is easy (the AI invents it before it reprograms itself to eliminate the bug). But I think this would be easier to get right, and what is the chance of time travel being easy compared to the chance of getting the "right" solution wrong?