I haven't been downvoting you, for what it's worth.
Anyway, I think our disagreement revolves around different interpretations of desirable in that quote (I think that definition's a little loose, incidentally, but that doesn't seem to be problematic here). You seem to be defining it as based on choice: a world-state is desirable relative to another if an agent would choose it over the other given the opportunity. That's pretty close to the thinking in economics among other disciplines, hence why I've been talking so much about revealed preference.
The problem is that we often choose things that turn out in retrospect to have been served our needs poorly. With that in mind I'm inclined to think of terminal values as irreducible terms in a utility function: features of future world-state that have a direct impact on an agent's well-being (a loose term, but hopefully an understandable one), and which can't be expressed in terms of more fundamental features. (There might be more than one decomposition of values here, in which case we should prefer the simplest one.)
That's fundamentally choice-agnostic, although elective concordance with outcomes might turn out to be such a term. Irrational risk aversion (though risk aversion can be rational, taking into account the limitations of foresight!) and other cognitive biases are features of choice, not of utility: if they worked on utility directly, we wouldn't call them biases.
By way of disclaimer, though, I should probably mention that this model isn't a perfect one when applied to humans: we don't seem to follow the VNM axioms consistently, so we can't be said to have utility functions in the strict sense. Some features of our cognition seem to behave similarly within certain bounds, though, and it's those that I'm focusing on above.
Excellently put, I think that sums up our disagreement very accurately. I'm not sure risk aversion couldn't be expressed as an irreducible term in a utility function, though. I suppose it would be more of a trait of the utility function, such as all probabilities are raised to a power greater than one, or something.
When I was a graduate student at the University of Notre Dame, I received a monthly living stipend of roughly $1,600. I decided to commit to giving ~10% of it to charity, and I had read in Peter Singer's book The Life You Can Save that one of the most efficient charities out there was Population Services International (PSI). Singer reported that GiveWell, a leading charity rating organization, had estimated that PSI's efforts saved lives at a cost of $650-$1000 each (pp. 88-89). So, I set up a recurring monthly donation of $160 to PSI, and kept it up for 15 months, for a total donation of $2,400.
I've been meaning to post the above information publicly for awhile, but was pushed over the edge by reading one of Eliezer's posts from a couple years back, Why Our Kind Can't Cooperate:
Since Eliezer's post is about rationalists, he stresses the issue of what arguments people voice. However, we know that just telling other people that you've given to charity makes them more likely to give. This is a point that Singer himself has emphasized.
I propose a thread for people to publicize their charitable donations. In light of the above, I'm especially interested to hear from people who've donated to the Singularity Institute for Artificial Intelligence. Once I acquire a regular source of income again in March, I intend to continue to primarily direct my charitable giving towards PSI, but maybe someone in this thread will persuade me to start giving to the Singularity Institute.