Humans desire fat and sugar. Those desires are built in - coded in genes.
That's a half-truth, or maybe a truth-value-less sentence. One could just as easily say humans desire calories and vitamin C.
Calories, yes, vitamin C - probably not. It took quite a while for the link between vitamin C deficiency and the foods containing it to be discovered. Humans apparently don't have an instinctive craving for it - perhaps because their diet is normally saturated with it.
Or perhaps humans simply desire survival and reproduction.
Sure - e.g. the maternal instinct.
I'm doubtful that any of these interpretations can claim to be the true one, at least until an individual human endorses one.
So: those are not really different interpretations of the same facts, but statements covering several different desires - so we don't have to choose between them.
It is better to regard chocolate gateau and ice cream as being learned associations with things actually valued.
"Actually valued" suggests that ice cream is not actually valued except as a means to fat and sugar, which is definitely not true. Just try taking away someone's ice cream and offering lard and sugar in their stead.
It was not an intended implication that fat and suger represent all the human gustatory desires.
We don't have to choose between statements of which desires are "coded in genes", but if we affirm too many of them we'll have more assumptions than are needed to explain the data. Why not just say that a purpose of the genes is to bring it about that in an appropriate environment the organism will consume adequate calories - rather than saying that the genes program a desire for fat? "Desire" is a psychological description first and foremost, and only incidentally, if at all, a term of evolutionary biology.
The jacket text for Keith Stanovich's The Robot's Rebellion sums up the book well:
The book is an excellent introduction to the first stage of Yudkowskian philosophy: We are robots in a mechanistic universe running on a swiss army knife of cognitive modules. But at least we finally noticed we're robots, and we can use the skills of rationality to hop off our habit treadmills and pursue our values instead. These values are complex and often arbitrary, but we can use our reflective capacities to extrapolate our values based on "higher-order" desires, a desire for preference consistency, and other considerations. All this is argued for at length in Stanovich's book. The only thing missing is a discussion of what to do about all this when AI arrives.