Ignoring the present problems with CEV, which are still deep and insufficiently understood to give any final judgment on that project, the relevant point is that CEV is supposed to solve the problem of existential threat of non-friendly AI, not to achieve improvements on the present human condition. In other words, it's an attempt to figure out how to ensure that an AI, if implemented, won't turn us into dog food, not a pseudoscientific recipe for building utopia here and now (which would be just as insane as all such previous ideas).
Assuming an AI will be implemented at some point, CEV will be a preferable alternative to being turned into dog food, and -- as a wild speculation -- in the hands of a superintelligence, its results might perhaps not even be that bad by other standards. But all this is extremely far-fetched in any case.
what's your problem with utopia? don't you like nice things?
Sam Harris has a new book, The Moral Landscape, in which he makes a very simple argument, at least when you express it in the terms we tend to use on LW: he says that a reasonable definition of moral behavior can (theoretically) be derived from our utility functions. Essentially, he's promoting the idea of coherent extrapolated volition, but without all the talk of strong AI.
He also argues that, while there are all sorts of tricky corner cases where we disagree about what we want, those are less common than they seem. Human utility functions are actually pretty similar; the disagreements seem bigger because we think about them more. When France passes laws against wearing a burqa in public, it's news. When people form an orderly line at the grocery store, nobody notices how neatly our goals and behavior have aligned. No newspaper will publish headlines about how many people are enjoying the pleasant weather. We take it for granted that human utility functions mostly agree with each other.
What surprises me, though, is how much flak Sam Harris has drawn for just saying this. There are people who say that there can not, in principle, be any right answer to moral questions. There are heavily religious people who say that there's only one right answer to moral questions, and it's all laid out in their holy book of choice. What I haven't heard, yet, are any well-reasoned objections that address what Harris is actually saying.
So, what do you think? I'll post some links so you can see what the author himself says about it:
"The Science of Good and Evil": An article arguing briefly for the book's main thesis.
Frequently asked questions: Definitely helps clarify some things.
TED talk about his book: I think he devotes most of this talk to telling us what he's not claiming.