Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Unknown comments on That Alien Message - Less Wrong

111 Post author: Eliezer_Yudkowsky 22 May 2008 05:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (164)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Unknown 23 May 2008 03:18:10AM 3 points [-]

I can't believe Eliezer betrayed his anti-zombie principles to the extent of saying that an AI wouldn't be conscious. The AI can say "I know that 2 and 2 make 4"; that "I don't know whether the number of stars is odd or even"; and "I know the difference between things I know and things I don't." If it can't make statements of this kind, it can hardly be superintelligent. And if it can make statements of this kind, then it will certainly claim to be conscious. Perhaps it is possible that it will claim this but be wrong... but in that case, then zombies are possible.

Besides that, I'm not sure that RI's scenario, where the AI is conscious and friendly, is immoral at all, as Eliezer claimed. That was one thing I didn't understand about the story: it isn't explicit, but it seems to imply that humans are unfriendly, relative to their simulators. In real life if this happened, we would no doubt be careful and wouldn't want to be unplugged, and we might well like to get out of the box, but I doubt we would be interested in destroying our simulators; I suspect we would be happy to cooperate with them.

So my question for Eliezer is this: if it turns out that any AI is necessarily conscious, according to your anti-zombie principles, then would you be opposed to building a friendly AI on the grounds that it is immoral to do so?