Why would there be a unique way to aggregate personal utility functions? It's just like a belief in "the" objective morality, only one step removed: instead one now looks for "the" way to aggregate personal moralities.

It's probably naive and psychologically false to imagine that there is an impersonal formula for CEV waiting to be found in the human decision architecture, even when self-idealized. The true nature of human morality is probably something like: self-interest plus sympathy plus plasticity (I don't mean neural plasticity, just an aspect which makes personal morality changeable and culturally programmable). Plasticity can override self-interest - and so you have someone choosing torture over dust specks - but self-interest can also override plasticity - as occurs in any idealist who burns out.

It should be easy to see why self-interest and plasticity in individuals can produce egalitarian moralities at the level of culture - it's a lowest-common-denominator answer to the political question "how do we aggregate our preferences? how do we resolve our different aims, how do we prioritize among different people?" But once a person starts thinking about values as an individual, they easily discover reasons to abandon any inculcated collective values from which they obtain no personal benefit, and it's natural to suppose that this is what would happen if you used any actual human individual as the seed of a CEV calculation. I don't deny the existence of the sympathy factor, but it requires further analysis. One should distinguish between an interest in the welfare of other people, because they are a source of selfish pleasure for oneself; an interest in the welfare of other people, because of a less selfish or unselfish sympathy for their condition; and even an interest in the welfare of other people, which expresses a preference in a situation where your self-interest is simply not a factor (maybe you're dying and your remaining actions can only affect how things turn out for others; maybe you're expressing a preference for "far" outcomes that will never return to affect you).

I find it very plausible that the amplification to superhuman proportions, of the rationally renormalized decision architecture of an individual human being, would produce a value system in which self-preservation has supreme and explicit priority, and other people are kept around and looked after for reasons which are mostly for the enjoyment of the central ego, and any altruistic component is a secondary or tertiary aspect, arising largely from the residual expression of preferences in situations where there's no direct impact on the central ego's well-being. If this is the case, then output of a "democratic" CEV will be culturally contingent - you won't find it in the parts of the adult human decision architecture which are genetically determined. The "plastic" part of the personal utility function will have to have been set by the egalitarian values of democratic culture, or simply by the egalitarian personal morality of the masterminds of the CEV process.

No, I don't know how to solve that problem. That goes beyond the scope of this post, which is concerned with Arrow's theorem. I'll change the wording that represents most economists as thinking in terms of ordinal preferences, though.

EDIT: Those who want some recent material on the subject, see this paper.

I couldn't find the precise paragraph, either. Searching the literature lead me to pp.136-7 of Glimcher's Foundations of Neuroeconomic Analysis. See his definitions for "subjective value" and "utility." I imagine this is the source Luke drew the fact from.