I've seen a bunch of different theories backed with varying amounts of experimental data - for instance, this, this and this - but I haven't looked at them enough to tell which ones seem most correct.

That said, I still don't remember running into any thorough discussion of what human preferences are, other than just "something that makes us make some choice in some situations". I mention here that

some of our preferences are implicit in our automatic habits (the things that we show we value with our daily routines), some in the preprocessing of sensory data that our brains carry out (the things and ideas that are ”painted with” positive associations or feelings), and some in the configuration of our executive processes (the actions we actually end up doing in response to novel or conflicting situations).

And I'm a little skeptical of any theory of human preferences that doesn't attempt to make any such breakdown and only takes a "black box" approach of looking at the outputs of our choice mechanism.