I agree with a twist: I think CEV will mostly be uninteresting, because it will have to integrate so many conflicting points of view that it will mostly come up with "do nothing at all".
Brainstorming a bit, I would say that value alignment is impossible unless an AI becomes actively part of the moral landscape: instead of being a slave to a hypothetical human uber-value, it will need to interact heavily with humans and force them to act so to reveal their true preferences or collaborate to generate a utopia.
I would also add that diversity of human values is value itself and important part of human culture. While it often results in conflicts and sufferings, homogenous New World may be even more unpleasant. CEV indirectly implies that there is one value system for all.
It is also based on orthogonality thesis which may be true for some AIs but not true for human brains. There is no separate values in human brain. "Value" is the way to describe human behaviour, but we can't show value neurons, or value texts in human brain. Our emotions, thoughts and memories are interconnected with our motivation and reward system, as well as all our cultural background. So we can't uploading values without uploading a human being.
If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "