continuity, personhood, and existence might well be illusions. If that is the case, my efforts to find ways to survive amount to extending something that isn't there in the first place
Can you say more about how you get from "X is an illusion" to "X isn't there in the first place"?
To clarify that question a little... suppose I'm thirsty in the desert, and am pursuing an image of water, and I eventually conclude to my disappointment that it is just a mirage.
I'm doing two things here:
Those are importantly different. If I were, instead, a non-thirsty student of optics, I would still correct my belief but I might not make the same value judgment: I might be delighted to discover that what I'd previously thought was a mere oasis is instead an interesting mirage!
In the same spirit, suppose I discover that continuity, personhood, and existence are illusions, when I had previously thought they were something else (what that "something else" is, I don't really know). So, OK, I correct my earlier false belief about the world.
There's still a value judgment left to make though... am I disappointed to realize I'm pursuing a mere illusion rather than the "something else" I actually wanted? Or am I delighted to discover that I'm pursuing a genuine illusion rather than an ill-defined "something else"?
Your way of speaking seems to take the former for granted. Why is that?
being less wrong past some threshold will not help us set better goals for ourselves
Well, it will, and it won't. But in the sense I think you mean it, yes, that's right... it won't.
Our values are what they are. Being less wrong improves our ability to implement those values, and our ability to articulate those values, which may in turn cause the values we're aware of and pursuing to become more consistent, but it doesn't somehow replace our values with superior values.
Let's say Bob's terminal value is to travel back in time and ride a dinosaur.
It is instrumentally rational for Bob to study physics so he can learn how to build a time machine. As he learns more physics, Bob realizes that his terminal value is not only utterly impossible but meaningless. By definition, someone in Bob's past riding a dinosaur is not a future evolution of the present Bob.
There are a number of ways to create the subjective experience of having gone into the past and ridden a dinosaur. But to Bob, it's not the same because he wanted both the subjective experience and the knowledge that it corresponded to objective fact. Without the latter, he might as well have just watched a movie or played a video game.
So if we took the original, innocent-of-physics Bob and somehow calculated his coherent extrapolated volition, we would end up with a Bob who has given up on time travel. The original Bob would not want to be this Bob.
But, how do we know that _anything_ we value won't similarly dissolve under sufficiently thorough deconstruction? Let's suppose for a minute that all "human values" are dangling units; that everything we want is as possible and makes as much sense as wanting to hear the sound of blue or taste the flavor of a prime number. What is the rational course of action in such a situation?
PS: If your response resembles "keep attempting to XXX anyway", please explain what privileges XXX over any number of other alternatives other than your current preference. Are you using some kind of pre-commitment strategy to a subset of your current goals? Do you now wish you had used the same strategy to precommit to goals you had when you were a toddler?