Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.

Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.

The study of affective synchrony shows that humans have simultaneously-active positive and negative affect systems. At extreme levels in either system, the other is shut down, but the rest of the time, they can support or oppose each other. (And in positions of opposition, we experience conflict and indecision.)

Meanwhile, the activation of these systems is influenced by current state/context/priming, as well as the envisioned future. So unless your attempt at modeling a utility function includes terms for all these things, you're sunk.

(Personally, this is where I think the idea of CEV has its biggest challenge: I know of no theoretical reason why humans must have convergent or consistent utility functions as individuals, let alone as a species.)