What if what they really want, deep down, is a sense of importance or social interaction or whatnot?

This sounds a bit like religious people saying "But what if it turns out that there is no morality? That would be bad!". What part of you thinks that this is bad? Because, that is what CEV is extrapolating. CEV is taking the deepest and most important values we have, and figuring out what to do next. You in principle couldn't care about anything else.

If human values wanted to self-modify, then CEV would recognise this. CEV wants to do what we want most, and this we call 'right'.

The only non-arbitrary "we" is the community of all minds/consciousnesses.

This is what you value, what you chose. Don't lose sight of invisible frameworks. If we're including all decision procedures, then why not computers too? This is part of the human intuition of 'fairness' and 'equality' too. Not the hamster's one.

When I talk with ecologists or environmentalists, almost always their reply is something like, "Yes, there's a lot of suffering, but it's okay because it's natural for them." One example:

As I sit here, thinking about the landscape of fear, I watch a small bird at my bird feeder. It spends more time looking around than it does eating. I try to imagine the world from its point of view — the startles, the alarms, the rustle of wings, the paw of the cat. And although I wish it well, I wouldn’t like its predators to disappear.

The argument seems to be less that the suffering is OK because it is natural than any intervention we can make to remove it would cause nature to not work, as in removing predator species results in more herbivores, which leads to vegetation being over consumed, which leads to ecological collapse. I am sympathetic to this argument. On a large enough scale, this means no breathable atmosphere. So while I think that wild animal suffering is a bad thing, I will accept it for now as a cost of supporting human life. (Maybe you could remove all animals not actually symbiotic with plants, but this seems like a hell of a gamble, we would likely regret the unintended consequences, and it could be difficult to undo.) Once humanity can upload and live in simulations, we have more options. Do you think the typical person advocating ecological balance has evaluated how the tradeoffs would change given future technology?

If more people thought about it harder, probably there would be more support, but ecological preservation is also a very strong intuition for some people. It's easy not to realize this when we're in our own bubbles of utilitarian-minded rationalists. :)

CEV is supposed to figure out what people would want if they were more rational. If rationalists tend to discard that intuition, it is not likely to have a strong effect on CEV. (Though if people without such strong intuitions are likely to become more rational, this would not be strong evidence. It may be useful to try raising the sanity waterline among people who demonstrate the intuition and see what happens.)

As far as insects, it's not obvious that post-humans would care enough to undertake the approximation of their brains that you mention, because maybe it would make the simulation more complicated (=> expensive) or reduce its fidelity.

I am completely against giving up the awesomeness of a good singularity because it is not obvious that the resulting society won't devote some tiny fraction of their computing power to simulations in which animals happen to suffer. The suffering is bad, but there are other values to consider here, that the scenario includes in far greater quantities.

CEV proponents can always avoid an unpalatable objection to CEV by saying that, "if we knew more and were smarter", we wouldn't act in such objectionable ways.

Yeah, it works well against naive objections that some humans behave badly and they have influence on the CEV.

But a reason has to be provided for thinking that greater intelligence or better knowledge would in fact cause us to act differently.

Note that I referred to "if we knew more and were smarter" after asking if proponents of preserving wild animal suffering have actually confronted the scale of suffering involved.

Just to make clear, are you saying that we should treat chickens how humans want to treat them, or how chickens do? Because if the former, then yeah, CEV can easily find out whether we'd want them to have good lives or not (and I think it would see we do).

But chickens don't (I think) have much of an ethical system, and if we incorporated their values into what CEV calculates, then we'd be left with some important human values, but also a lot of chicken feed.

Objective? Sure, without being universal.

Human beings are physically/genetically/mentally similar within certain tolerances; this implies there is one system of ethics (within certain tolerances) that is best suited all of us, which could be objectively determined by a thorough and competent enough analysis of humans. The edges of the bell curve on various factors might have certain variances. There might be a multi-modal distribution of fit (bimodal on men and women, for example), too. But, basically, one objective ethics for humans.

This ethics would clearly be unsuited for cats, sharks, bees, or trees. It seems vanishingly unlikely that sapient minds from other evolutions would also be suited for such an ethics, either. So it's not universal, it's not a code God wrote into everything. It's just the best way to be a human . . . as humans exposed to it would in fact judge, because it's fitted to us better than any of our current fumbling attempts.

The CEV of humanity is not likely to promote animal suffering. Most people don't value animal suffering. They value eating hamburgers, and aren't particularly bothered by the far away animal suffering that makes it possible for them to eat hamburgers. An FAI can give us hamburgers without causing any animal suffering.

Knowing that you've abandoned moral realism, how would you respond to someone making an analogous argument about preferences or duties? For instance, "When a preference of mine is frustrated, I come to see this as a state of affairs that ought not to exist," or "When someone violates a duty, I come to see this as a state of affairs that ought not to exist." Granted, the acquaintance may not be as direct as in the case of intense suffering. But is that enough to single out pleasure and suffering?

As I noted on the 80,000 Hours thread, intermediaries are nearly always an added expense on the distribution side. In this case, distribution of donations. The immediate impact is that fewer donation dollars (or whatever currency) find their way to the target organizations. The exception is if an intermediate organization facilitates a 100% pass-through, due to other funding or altruistic efforts.

Since you don't want to sound narcissistic, consider counting the number of times you use "I" "my" and "me" before posting. Also avoid the temptation to share your whole mental process and focus on conclusions and supporting evidence that moves beyond personal anecdote. Don't defend yourself, defend your ideas.

Three friends independently pointed me to Overcoming Bias in fall/winter 2006.