JoshuaZ comments on Sarah Connor and Existential Risk - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (77)
I'd be more interested in a response to the substance of my comment: If you think that a person is about to turn on a (to your way of thinking) insufficiently Friendly AI, such that killing them might stop the inevitable paperclipping of all you hold dear, how do you take into account the fact that they might have outwitted you by setting up a dead man's switch?
In other words, how do you take into account the fact that killing them might bring about exactly the fate that you intend to prevent; whereas one more exchange of rational argument might convince them not to do it?
Edit: Disregard what I've wrote below. It isn't relevant since it assumes that the individual hasn't tried to make a Friendly AI which seems to be against the assumption in the hypothetical.
There seems to be a heavy overlap between people who think AGI will foom and people who are concerned about Friendliness (for somewhat obvious reasons. Friendliness matters a lot more if fooming is plausible). It seems extremely unlikely that someone would set up a dead man's switch unless they thought that a lot would actually get accomplished by the AI, i.e. that it would likely foom in a Friendly fashion. The actual chance that any such switches have been put into place seems low.
Oh, sure, I agree.
But what if Eliezer thinks he's got an FAI he can turn on, and Joe isn't convinced that it's actually as Friendly as Eliezer thinks it is? I'd rather Joe argue with Eliezer than shoot him.