My preferences are a very delicate issue that can be easily disappointed. I can't change reality so I simply have to live with the fact that nothing can travel faster than light even though I would love to (maybe not, just an example). The same might be true for CEV, it might be the only option. But I would love to see a different future, one with more adventure and struggle, something more akin to Orion's Arm.
And no, choosing to just forget about the CEV process wouldn't be what I wanted either. Just imagine an alien came along offering you its help, you'd hardly ask it to leave and make you forget that it does exist.
Is reading Fantasy, and thinking about alternate impossible universe right now "just stupid"? Or studying mythology for that matter?
No, but I personally don't like fantasy. Science fiction still has some appearance of plausibility, which would be removed under CEV.
Is someone playing chess just stupid, because they wouldn't stand a chance against Kasparov or Deep Blue?
No, I quite enjoy playing games that I know that I can't win. But under CEV that would mainly be a result of choice, if humanity wanted it then we would become all equal.
...ask fans of novels whether they're curious about the end of said novels (especially mysteries), even though they have the capacity to flip to the end.
This seems similar to the argument made by people about death, "every song ends, but is that any reason not to enjoy the music?" And a song ends doesn't stop me from listening to music, but knowing that my life would end in a week or 10 years would pretty much destroy any fun that I could have in the remaining time. As I said, my preferences seem to be complex and CEV could easily destroy the delicate balance.
It would still be awesome to explore nature and play games of course. But I am pretty sure that people wouldn't build particle accelerators if they could just "flip to the end" and read about the theory of everything. Or even if they would do so, a lot of the previous excitement, that was result of the possibility to discover something novel, would be gone. After all, nobody writes papers about figuring out the plot of a fiction story and receives a Nobel prize for it.
Suppose we want to use the convergence of humanity's preferences as the utility function of a seed AI that is about to determine the future of its light cone.
We figured out how to get an AI to extract preferences from human behavior and brain activity. The AI figured out how to extrapolate those values. But my values and your values and Sarah Palin's values aren't fully converging in the simulation running the extrapolation algorithm. Our simulated beliefs are converging because on the path to reflective equilibrium our partially simulated selves have become true Bayesians and Aumann's Agreement Theorem holds. But our preferences aren't converging quite so well.
What to do? We'd like the final utility function in the FOOMed AI to adhere to some common-sense criteria:
Now, Arrow's impossibility theorem says that we can only get the FOOMed AI's utility function to adhere to these criteria if the extrapolated preferences of each partially simulated agent are related to each other cardinally ("A is 2.3x better than B!") instead of ordinally ("A is better than B, and that's all I can say").
Now, if you're an old-school ordinalist about preferences, you might be worried. Ever since Vilfredo Pareto pointed out that cardinal models of a person's preferences go far beyond our behavioral data and that as far as we can tell utility has "no natural units," some economists have tended to assume that, in our models of human preferences, preference must be represented ordinally and not cardinally.
But if you're keeping up with the latest cognitive neuroscience, you might not be quite so worried. It turns out that preferences are encoded cardinally after all, and they do have a natural unit: action potentials per second. With cardinally encoded preferences, we can develop a utility function that represents our preferences and adheres to the common-sense criteria listed above.
Whaddya know? The last decade of cognitive neuroscience has produced a somewhat interesting result concerning the plausibility of CEV.