My values seem to be pinned on these ideas (the ones that are not true) because while I am in the process of caring about the things I care about, and especially when I am making a choice about something, I find that I am always making the assumption that these ideas are true -- that the states of the universe matter and that I exist forever.
When it occurs to me to remember that these assumptions are not true, I feel a great deal of cognitive dissonance. However, the cognitive dissonance has no resolution. I think about it for a little while, go about my business, and discover some time later I forgot again.
I don't know if a specific example will help or not. I am driving home, in traffic, and brain is happily buzzing with thoughts. I am thinking about all the people in cars around me and how I'm part of a huge social network and whether the traffic is as efficient as it could be and civilization and how I am going to go home and what I am going to do. And then I remember about death, the snuffing out of my awareness, and something about that just doesn't connect. It's like I can empathize with my own non-existence (hopefully this example is something more than just a moment of psychological disorder) and I feel that my current existence is a mirage. Or rather, the moral weight that I've given it doesn't make sense. That's what the cognitive dissonance feels like.
Let's say Bob's terminal value is to travel back in time and ride a dinosaur.
It is instrumentally rational for Bob to study physics so he can learn how to build a time machine. As he learns more physics, Bob realizes that his terminal value is not only utterly impossible but meaningless. By definition, someone in Bob's past riding a dinosaur is not a future evolution of the present Bob.
There are a number of ways to create the subjective experience of having gone into the past and ridden a dinosaur. But to Bob, it's not the same because he wanted both the subjective experience and the knowledge that it corresponded to objective fact. Without the latter, he might as well have just watched a movie or played a video game.
So if we took the original, innocent-of-physics Bob and somehow calculated his coherent extrapolated volition, we would end up with a Bob who has given up on time travel. The original Bob would not want to be this Bob.
But, how do we know that _anything_ we value won't similarly dissolve under sufficiently thorough deconstruction? Let's suppose for a minute that all "human values" are dangling units; that everything we want is as possible and makes as much sense as wanting to hear the sound of blue or taste the flavor of a prime number. What is the rational course of action in such a situation?
PS: If your response resembles "keep attempting to XXX anyway", please explain what privileges XXX over any number of other alternatives other than your current preference. Are you using some kind of pre-commitment strategy to a subset of your current goals? Do you now wish you had used the same strategy to precommit to goals you had when you were a toddler?