The Standard PD is set up so there are only two agents and only their choices and values matter. I tend to think of rationality in these dilemmas as being largely a matter of reputation, even when the situation is circumscribed and described as one-shot. Hofstadter's concept of super-rationality is part of how I think about this. If I have a reputation as someone who cooperates when that's the game-theoretically optimal thing to do, then it's more likely that whoever I've been partnered with will expect that from me, and cooperate if he understands why that strategy works.
Since it would buttress that reputation, I keep hoping that rationalists, generally, would come to embrace some interpretation of super-rationality, but I keep seeing self-professed rationalists whose choices seem short-sightedly instrumentalist to me.
But this seems to be a completely different situation. Rather than attempting to cooperate with someone who I should assume to be my partner, and who has my interests at heart, I'm asked to play a game with someone who doesn't reason the way I do, and who explicitly mistrusts my reasoning. In addition, the payoff isn't to me and the other player, the payoff is to a huge number of uninvolved other people. MBlume seems to want me to think of it in terms of something valuable in my preference ranking, but he's actually set it up so that it's not a prisoner's dilemma, it's a hostage situation in which I have a clearly superior choice, and an opportunity to try to convince someone whose reasoning is alien to my own.
I defect. I do my best to convince my friend that the stakes are too high to justify declaring his belief in god. So you can get me to defect, but only by setting up a situation in which my allies aren't sitting on the other side of the bargaining table.
I spoke yesterday of the epistemic prisoner's dilemma, and JGWeissman wrote:
To which I said:
And lo, JGWeissman saved me a lot of writing when he replied thus:
I make one small modification. You and your creationist friend are actually not that concerned about money, being distracted by the massive meteor about to strike the earth from an unknown direction. Fortunately, Omega is promising to protect limited portions of the globe, based on your decisions (I think you've all seen enough PDs that I can leave the numbers as an excercise).
It is this then which I call the true epistemic prisoner's dilemma. If I tell you a story about two doctors, even if I tell you to put yourself in the shoes of one, and not the other, it is easy for you to take yourself outside them, see the symmetry and say "the doctors should cooperate". I hope I have now broken some of that emotional symmetry.
As Omega lead the creationist to the other room, you would (I know I certainly would) make a convulsive effort to convince him of the truth of evolution. Despite every pointless, futile argument you've ever had in an IRC room or a YouTube thread, you would struggle desperately, calling out every half-remembered fragment of Dawkins or Sagan you could muster, in hope that just before the door shut, the creationist would hold it open and say "You're right, I was wrong. You defect, I'll cooperate -- let's save the world together."
But of course, you would fail. And the door would shut, and you would grit your teeth, and curse 2000 years of screamingly bad epistemic hygiene, and weep bitterly for the people who might die in a few hours because of your counterpart's ignorance. And then -- I hope -- you would cooperate.