As I understand it, there are separate taste receptors for fat and sugar - so it is probably better to say that humans desire some types of fat and sugar than to say that they desire calories.
And by the same token, it is probably even better to say that they desire ice cream and/or the taste of ice cream, and so on for other particular foods.
So: my position is that it is fine to talk like that - provided one makes the distinction between proximate and ultimate values. There's a pretty neat and general way of abstracting learning systems out into agent, ultimate values and environment using the framework of reinforcement learning. Under that abstraction, "the taste of ice cream" is not one of the ultimate values. Those values might include diversity, contrast and texture as well as fat and sugar - but I don't think there's much of a case for putting "the taste of ice cream" in there.
Genes built our desires, but their "purposes" in doing so are not identical to those desires.
I think I already acknowledged that distinction - with my example of "taking rewarding drugs" being something that the brain wants, but the genes do not.
Whereas, in the context of our conversation here, it would not be too wrong to say that humans' purposes are our desires.
Maybe - depending on which parts of yourself you most identify with.
There's a pretty neat and general way of abstracting learning systems out into agent, ultimate values and environment using the framework of reinforcement learning.
Interesting. I'd appreciate references or links. To me, the interesting and still open question is how these "ultimate" values relate to the outcome of rational reflection and experimentation by the individual.
The jacket text for Keith Stanovich's The Robot's Rebellion sums up the book well:
The book is an excellent introduction to the first stage of Yudkowskian philosophy: We are robots in a mechanistic universe running on a swiss army knife of cognitive modules. But at least we finally noticed we're robots, and we can use the skills of rationality to hop off our habit treadmills and pursue our values instead. These values are complex and often arbitrary, but we can use our reflective capacities to extrapolate our values based on "higher-order" desires, a desire for preference consistency, and other considerations. All this is argued for at length in Stanovich's book. The only thing missing is a discussion of what to do about all this when AI arrives.