It's not just CEV and recursive self-modification, either. CEV only works on individuals and (many) individuals will FOOM once they acquire FAI. If individuals don't FOOM into FAI's (and I see no reason that they would choose to do so) we need a fully general moral/political theory that individuals can use to cooperate in a post-singularity world. How does ownership work when individuals suddenly have the ability to manipulate matter and energy on much greater scales than even governments can today? Can individuals clone themselves in unlimited number? Who actually owns solar and interstellar resources? I may trust a FAI to implement CEV for an individual but I don't necessarily trust it to implement a fair universal political system; that's asking me to put too much faith in a process that won't have sufficient input until it's too late. If every individual FOOMed at the same rate perhaps CEV could be used over all of humanity to derive a fair political system, but that situation seems highly unlikely. The most advanced individuals will want and need the most obscure and strange sounding things and current humans will simply be unable to fully evaluate their requests and the implications of agreeing to them.
I think we have the same sentiment, though we may be using different terminology. To paraphrase Eliezer on CEV, "Not to trust the self of this passing moment, but to try to extrapolate a me who knew more, thought faster, and were more the person I wished I were. Such a person might be able to avoid the fundamental errors. And still fearful that I bore the stamp of my mistakes, I should include all of the world in my extrapolation." Basically, I believe there is no CEV except the entire whole of human morality. Though I do admit that CEV has ...
Related Posts: A cynical explanation for why rationalists worry about FAI, A belief propagation graph
Lately I've been pondering the fact that while there are many critics of SIAI and its plan to form a team to build FAI, few of us seem to agree on what SIAI or we should do instead. Here are some of the alternative suggestions offered so far: