This logic assumes that a beyond human intelligence in a redesigned world would still find inherent value in free will. Isn't it possible that such an intelligence would move beyond the need to experience pain in order to comprehend the value of pleasure?
According to the bible, god created different aspects of the world across six days and after each creation he "saw that it was good". Yet nothing ELSE existed. If there had never been a "world" before, and evil had not yet been unleashed, by what method was this god able to measure that his creation was good? One must assume that god's superior intelligence simply KNEW it to be good and had no need to measure it against something "bad" in order to know it. Couldn't the eventual result of AI be the attainment of the same ability... the ability to KNOW pleasure without the existence of its opposite?
Isn't the hope (or should I say fun?) of considering the potential of AI that such a vast intelligence would move life BEYOND the anchors to which we now find ourselves locked? If AI is simply going to be filled with the same needs and methods of measuring "happiness" as we currently deal with, what is the point of hoping for it at all?
This is a bit of an afterthought, but even at our current level of intelligence, humans have no way of knowing if we would value pleasure if pain did not exist. Pain does now and has always existed. "Evil" (or what we perceive as evil) has existed since the dawn of recorded human existence. How can we assume that we are not already capable of recognizing pleasure as pleasure and good as good without their opposites to compare them to? We have never had the opportunity to try.
I beg to differ on the aspect of there being non-existence predating the creation. A subtle nuance in the first verse of Genesis offers an insight into this. Gen 1:1 "In the beginning God created the heavens and the earth. And the earth was without form, and void; and darkness was upon the face of the deep." Original manuscripts offer a translation that is closer to "and the earth 'became' without form (sic), and void". It may so very well be that in the assumption that God looked on his creation and saw that it was good, there was a p...
(A shorter gloss of Fun Theory is "31 Laws of Fun", which summarizes the advice of Fun Theory to would-be Eutopian authors and futurists.)
Fun Theory is the field of knowledge that deals in questions such as "How much fun is there in the universe?", "Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having more fun?"
Many critics (including George Orwell) have commented on the inability of authors to imagine Utopias where anyone would actually want to live. If no one can imagine a Future where anyone would want to live, that may drain off motivation to work on the project. The prospect of endless boredom is routinely fielded by conservatives as a knockdown argument against research on lifespan extension, against cryonics, against all transhumanism, and occasionally against the entire Enlightenment ideal of a better future.
Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil). Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance. Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization. Fun Theory also highlights the flaws of any particular religion's perfect afterlife - you wouldn't want to go to their Heaven.
Finally, going into the details of Fun Theory helps you see that eudaimonia is complicated - that there are many properties which contribute to a life worth living. Which helps you appreciate just how worthless a galaxy would end up looking (with very high probability) if the galaxy was optimized by something with a utility function rolled up at random. This is part of the Complexity of Value Thesis and supplies motivation to create AIs with precisely chosen goal systems (Friendly AI).
Fun Theory is built on top of the naturalistic metaethics summarized in Joy in the Merely Good; as such, its arguments ground in "On reflection, don't you think this is what you would actually want for yourself and others?"
Posts in the Fun Theory sequence (reorganized by topic, not necessarily in the original chronological order):