We need another tournament!
With humanity and the Paperclipper facing 100 rounds of the iterated Prisoner's Dilemma, do you really truly think that the rational thing for both parties to do, is steadily defect against each other for the next 100 rounds?
No, because the rationalist thing to do is not to assume that the other player is a perfect rationalist; or, to be more precise, that the other player is a perfect rationalist who assumes that you're a perfect rationalist who assumes that the other player is a perfect rationalist who...
Defecting the first round only makes sense if you assume that the other player also defects that round, because the benefit from mutual cooperation is much larger than 1 million human lives (the difference between CC and DD), since it goes without saying that DD will lead to DD in the next round, while CC may lead to another CC, and another CC, and so on, even if not until the end.
IPD is not about defeating the "opponent", it is about maximizing points (or lives saved or whatever). If your opponent defects every round, and you cooperate the first round, you effectively save one million less lives than you reasonably could have. If you always defect, and your opponent plays TFT, you "save" 1 million additional lives in the first round—and lose 1 million for each round your opponent would have cooperated. If you expect your opponent to either always defect, or to play some kind of TFT-nD (that is, tit for tat and defect the last n rounds with unknown n>0), then by defecting the first round you save 100 million respectively 101 million lives, while by not defecting first you save 99 million respectively 199-n million lives. That means that for you to rationally defect the first round, you must have very, very high confidence in your opponent not playing some sort of TFT-nD with low n.
If I know this is your strategy, I don't have to be a perfect rationalist to know that I can defect on the first round.
Today's post, The Truly Iterated Prisoner's Dilemma was originally published on 04 September 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The True Prisoner's Dilemma, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.