Anticipation and faith are both aspects of the human decision process, in a sense just subroutines of a larger program, but they also generate subjective experiences (qualia) that we value for their own sake. Suppose you ask a religious friend why he doesn’t give up religion, he might say something like “Having faith in God comforts me and I think it is a central part of the human experience. Intellectually I know it’s irrational, but I want to keep my faith anyway. My friends and the government will protect me from making any truly serious mistakes as a result of having too much faith (like falling into dangerous cults or refusing to give medical treatment to my children)."
Personally I've never been religious, so this is just a guess of what someone might say. But these are the kinds of thoughts I have when faced with the prospect of giving up the anticipation of future experiences (after being prompted by Dan Armak). We don't know for sure yet that anticipation is irrational, but it's hard to see how it can be patched up to work in an environment where mind copying and merging are possible, and in the mean time, we have a decision theory (UDT) that seems to work fine, but does not involve any notion of anticipation.
What would you do if true rationality requires giving up something even more fundamental to the human experience than faith? I wonder if anyone is actually willing to take this step, or is this the limit of human rationality, the end of a short journey across the space of possible minds?
I don't see any reason why you should give up anticipation of future experiences. It's possible that in situations involving duplication and merging of minds you should accept that anticipation is an unreliable guide for decision-making, but that's not at all the same thing. (It's more like a religious person agreeing that if he gets seriously ill he should see a doctor rather than relying on his god to cure him -- which, in fact, religious people generally do.) At least for the present, the sort of switching and copying and merging and muddling of minds that would make our usual anticipation-based thinking fail badly doesn't happen to any appreciable extent.