I refer, of course, to people whose preferences really are different to our own. Coherent Extrapolated Assholes. I don't refer to people who would really have preferences that I would consider acceptable if they just knew a bit more.

You asked for an explanation of how a correctly implemented 'CEV' could want something abhorrent. That's how.

There is an unfortunate tendency to glorify the extrapolation process and pretend that it makes any given individual or group have acceptable values. It need not.

Punishment from the future is spooky enough. Imagine what an anti-Guns of the South would be like for the temporal locals. Not pleasant, that's for sure.

Doesn't CEV implicitly assert that there exists a set of moral assertions M that is more reliably moral than anything humans assert today, and that it's possible for a sufficiently intelligent system to derive M?

That sure sounds like a belief in moral progress to me.

Granted, it doesn't imply that humans left to their own devices will achieve moral progress. But the same is true of technological progress.

If I do not want my preferences to be altered in the kind of way you mention then a Friendly (to me) AI doesn't do them.

I just don't see how that is possible without the AI becoming a primary attractor and therefore fundamentally altering the trajectory of your preferences. I'd favor the way Kurzweil portrays a technological Singularity here, where humans themselves become the Gods. I do not want to live in a universe where I'm just a puppet of the seed I once sowed. That is, I want to implement my own volition without the oversight of a caretaker God. As long as there is a being vastly superior to me that takes interest in my own matters, even the mere observer effect will alter my preferences since I'd have to take this being into account in everything conceivable.

The whole idea of friendly AI, even if it was created to suit only my personal volition, reminds me of the promises of the old religions. This horrible boring universe where nothing bad can happen to you and everything is already figured out by this one being. Sure, it wouldn't figure it out if it knew I want to do that myself. But that be pretty dumb, as it could if I wanted it to. And that's just the case with my personal friendly AI. One based on the extrapolated volition of humanity would very likely not be friendly towards me and would ultimately dictate what I can and cannot do.

Really the only favorable possibility here is to merge with the AI. But that would mean instant annihilation to me as I would add nothing to a being that vast. So I still hope that AI going foom is wrong and that we see a slow development over many centuries instead, without any singularity type event.

And I'm aware that big government and other environmental influences are altering and stearing my preferences as well. But they are much more fuzzy whereas a friendly AI is very specific. The more specific, the less free will I do have. That is, the higher the ratio of influence and effectiveness of control that I exert over the environment to the environment over me the more free I am to implement what I want to do versus what others want me to do.