As such, we'd be unlikely to get what we really want if the world was re-engineered in accordance with a description of what we want that came from verbal introspective access to our motivations.

Interesting as these experimental results are, it sounds to me like you're saying that there's a license to be human (or a license to be yourself, or a license to be your current self).

Suppose I found out that many of my actions that seemed random were actually subtly aimed at invading Moldova, perhaps because aliens with weird preferences placed some functional equivalent of mind control lasers in my brain, and suppose that this fact was not introspectively accessible to me; e.g., a future where Moldova is invaded does not feel more utopian to imagine than the alternatives. Isn't there an important sense in which, in that hypothetical, I don't care about invading Moldova? What if the mind control laser was outside my brain, perhaps in orbit? At what point do I get to say, "I won't let my so-called preferences stop me from doing what's right?"

My impression is that this mindset, where you determine what to do by looking closely at the world to see what you're already doing, and then giving that precedence over what seems right, would be seen as an alien mindset by anyone not affected by certain subtle misunderstandings of the exact sense in which value is subjective. My impression is that once these misunderstandings go away and people ask themselves what considerations they're really moved by, they'll find out that where their utility function (or preferences or whatever) disagrees with what, on reflection, seems right, they genuinely don't care (at least in any straightforward way) what their preferences are, paradoxical as that sounds.

Or am I somehow confused here?