there can be an irrational difference between what we believe our values are and what they really are.

Certainly. We are not transparent to ourselves: we have subconscious and situation-dependent drives; we don't know in advance precisely how we'll respond to hypothetical situations, how much we'll enjoy and value them; we have various biases and inaccurate/fake memory issues which cause us to value things wrongly because we incorrectly remember enjoying them; our conscious selves self-deceive and are deceived by other brain modules; and so on.

Moreover, humans don't have well-defined (or definable) utility functions; our different values conflict.