I am saying that a CEV that extrapolated human morality would generally be utilitarian, but that it would grant a utility value of zero to satisfying what I call "malicious preferences." That is, if someone valued frustrating someone else's desires purely for their own sake, not because they needed the resources that person was using or something like that, the AI would not fulfill it.

This is because I think that a CEV of human morality would find the concept of malicious preferences to be immoral and discard or suppress it. My thinking on this was inspired by reading about Bryan Caplan's debate with Robin Hanson, where Bryan mentioned:

...Robin endorses an endless list of bizarre moral claims. For example, he recently told me that "the main problem" with the Holocaust was that there weren't enough Nazis! After all, if there had been six trillion Nazis willing to pay $1 each to make the Holocaust happen, and a mere six million Jews willing to pay $100,000 each to prevent it, the Holocaust would have generated $5.4 trillion worth of consumers surplus.

I don't often agree with Bryan's intuitionist approach to ethics, but I think he made a good point, satisfying the preferences of those trillion Nazis doesn't seem like part of the meaning of right, and I think a CEV of human ethics would reflect this. I think that the preference of the six million Jews to live should be respected and the preferences of the six trillion Nazis be ignored.

I don't think this is because of scope insensitivity, or because I am not a utilitarian. I endorse utilitarian ethics for the most part, but think that "malicious preferences" have zero or negative utility in their satisfaction, no matter how many people have them. For conflicts of preferences that involve things like disputes over use of scarce resources, normal utilitarianism applies.

In response to your question I have edited my post and changed "a proper CEV" to "a CEV of human morality."