As I understand it, to the extent that it makes sense to cooperate, the thing that cooperates is not you, but some sub-algorithm implemented in both you and your opponent.
It has to add up to normality, there should be a you somewhere. If each time you act on your better judgment over gut instinct it is "not you" that does the acting, why is it invited in your mind? Is the whole of deliberate reasoning not you?
In my book, when future-you fights a previously made informed commitment, then it is a case where future-you is not you anymore, where it stops caring about your counterfactuals. Not when the future-you remains reflectively consistent.
But possibly, this reflectively consistent creature can't a person anymore, and is not what we'd like to be, with our cognitive ritual morally significant after all, a thing to protect in itself.
I spoke yesterday of the epistemic prisoner's dilemma, and JGWeissman wrote:
To which I said:
And lo, JGWeissman saved me a lot of writing when he replied thus:
I make one small modification. You and your creationist friend are actually not that concerned about money, being distracted by the massive meteor about to strike the earth from an unknown direction. Fortunately, Omega is promising to protect limited portions of the globe, based on your decisions (I think you've all seen enough PDs that I can leave the numbers as an excercise).
It is this then which I call the true epistemic prisoner's dilemma. If I tell you a story about two doctors, even if I tell you to put yourself in the shoes of one, and not the other, it is easy for you to take yourself outside them, see the symmetry and say "the doctors should cooperate". I hope I have now broken some of that emotional symmetry.
As Omega lead the creationist to the other room, you would (I know I certainly would) make a convulsive effort to convince him of the truth of evolution. Despite every pointless, futile argument you've ever had in an IRC room or a YouTube thread, you would struggle desperately, calling out every half-remembered fragment of Dawkins or Sagan you could muster, in hope that just before the door shut, the creationist would hold it open and say "You're right, I was wrong. You defect, I'll cooperate -- let's save the world together."
But of course, you would fail. And the door would shut, and you would grit your teeth, and curse 2000 years of screamingly bad epistemic hygiene, and weep bitterly for the people who might die in a few hours because of your counterpart's ignorance. And then -- I hope -- you would cooperate.