Assumption 1: The paperclip maximizer does not have magical information about my intentions, so changing my intentions will not change its decision.
I prefer (D,C) to (C,C), and I prefer (D,D) to (C,D). If I were to know the paperclip maximizer's choice ahead of time, it would not change my choice; and if the paperclip maximizer knew my choice ahead of time, it would not change its choice.
Here's the real question: Are you willing to sacrifice two paperclips to save a billion human lives? Is the paperclip maximizer willing to sacrifice two billion human lives to create one paperclip?
Would it be rational to execute the reverse, even if possible: For the humans to sacrifice human lives to create paperclips, on the possibility that the paperclip maximizer would sacrifice paperclips to save humans? Given that they are better at saving humans than we are, and that we are better at making paperclips than they are, it would appear to be the case.
There's no given reason why you can't just defect and then make four paperclips so it's better for the AI. For that matter, the humans saved will need paperclips, so net good for the AI.
I think it would work if you specify that the AI only cares about paperclips in its own universe.
Today's post, The True Prisoner's Dilemma was originally published on 03 September 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Dreams of Friendliness, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.