everything we want is as possible and makes as much sense as wanting to hear the sound of blue or taste the flavor of a prime number

We know it isn't because most of the time we get what we want. You want chocolate, so you go and buy some and then eat it, and the yummy chocolatey taste you experience is proof that it wasn't that futile after all for you to want chocolate.

The feeling of reward we get when we satisfy some of our terminal values is what makes us think that they aren't based on a misunderstanding of reality. So it's probably a pretty good bet to keep wanting at least the things that have led to rewards in the past, even if we aren't as sure about the rest of them, like going back in time.