Aumann's agreement theorem seems to imply that individual extrapolated volition assignments must agree on statements of fact: the setup implies that they're simulated as perfect reasoners and share a knowledge pool, and the extrapolation process provides for an unbounded number of Bayesian updates. So we can expect extrapolated volition to cohere exactly to the extent that it's based on common fundamental goals: not immediate desires and not possibly-fallible philosophical results, but the low-level affective assignments that lead us to think of those higher-level results as desirable or undesirable.
To what extent do common fundamental goals drive our moral reasoning? I don't know, but individual differences do exist (the existence of masochistic people should prove that), and if they're large enough then CEV may end up looking incomplete or unpleasantly compromise-driven.
I don't know, but individual differences do exist (the existence of masochistic people should prove that)
But is that relevant to the question that CEV tries to answer? As far as I know, most masochistic people don't also hold a belief that everybody should be masochistic.
Taken from some old comments of mine that never did get a satisfactory answer.
1) One of the justifications for CEV was that extrapolating from an American in the 21st century and from Archimedes of Syracuse should give similar results. This seems to assume that change in human values over time is mostly "progress" rather than drift. Do we have any evidence for that, except saying that our modern values are "good" according to themselves, so whatever historical process led to them must have been "progress"?
2) How can anyone sincerely want to build an AI that fulfills anything except their own current, personal volition? If Eliezer wants the the AI to look at humanity and infer its best wishes for the future, why can't he task it with looking at himself and inferring his best idea to fulfill humanity's wishes? Why must this particular thing be spelled out in a document like CEV and not left to the mysterious magic of "intelligence", and what other such things are there?