I response to my own question: I think that the information difference between innate biological prefs that we have and explicitly stated preferences is a lot bigger than I thought.
For example, I can state the following:
(1) I wish to be smart enough to understand all human science and mathematics published to this date, and to solve all outstanding scientific and philosophical questions including intelligence, free will and ethics. I want to know the contents and meaning of every major literary work in print and every major film, to understand the history of every major civilization, to fall in love with the person who is most compatible with me in the world.
Now if I make all these wishes, how much have I cut down future states of the universe? How much optimizing power in bits have I wished for?
I expressed the wish in about 330 characters, which according to Shannon means I have expressed 330 bits of information, roughly equivalent to specifying the state of a 20X20 grid of pixels each one of which can be either on or off. I feel that this is something of an underestimate in terms of how much I have cut down future states of the universe. Another way of calculating the complexity of the above wish is to bound it by the log of the number of psychologically distinguishable states of my mind. Given the FHI brain emulation roadmap, this upper bound could be a very large number indeed. Here is another ~300-char wish:
(2) I want to be as rich as Bill Gates. I want to have ten mansions, each with ten swimming pools and a hundred young, willing female virgins to cater to my every whim. I want my own private army and an opposing force who I will trounce in real combat every weekend. I want an 18-inch penis and muscles the size of Arnie in his younger days, and I want to be 6'7''. I want to be able to eat galaxy chocolate flavored ice cream all day without getting fat or getting bored with it. I want a car that goes at 5000 miles an hour without any handling problems or danger of accident, and I want to be able to drive it around the streets of my city and leave everyone in the dust.
Now it appears to me that this wish probably did only cut down the future by 300 bits... that it is a far less complex wish than the first one I gave. Presumably the difference between those who end up in low grade heaven and those who end up as superintelligent posthumans inhabiting a Dyson sphere, or having completely escaped from our physics lies in the difference between the former wish and the latter. Again, it is fruitful and IMO very important to explore the continuum between the two.
(A shorter gloss of Fun Theory is "31 Laws of Fun", which summarizes the advice of Fun Theory to would-be Eutopian authors and futurists.)
Fun Theory is the field of knowledge that deals in questions such as "How much fun is there in the universe?", "Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having more fun?"
Many critics (including George Orwell) have commented on the inability of authors to imagine Utopias where anyone would actually want to live. If no one can imagine a Future where anyone would want to live, that may drain off motivation to work on the project. The prospect of endless boredom is routinely fielded by conservatives as a knockdown argument against research on lifespan extension, against cryonics, against all transhumanism, and occasionally against the entire Enlightenment ideal of a better future.
Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil). Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance. Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization. Fun Theory also highlights the flaws of any particular religion's perfect afterlife - you wouldn't want to go to their Heaven.
Finally, going into the details of Fun Theory helps you see that eudaimonia is complicated - that there are many properties which contribute to a life worth living. Which helps you appreciate just how worthless a galaxy would end up looking (with very high probability) if the galaxy was optimized by something with a utility function rolled up at random. This is part of the Complexity of Value Thesis and supplies motivation to create AIs with precisely chosen goal systems (Friendly AI).
Fun Theory is built on top of the naturalistic metaethics summarized in Joy in the Merely Good; as such, its arguments ground in "On reflection, don't you think this is what you would actually want for yourself and others?"
Posts in the Fun Theory sequence (reorganized by topic, not necessarily in the original chronological order):