If we take for granted that an AI that is friendly to all potential creatures is out of the question - that the only type of FAI we really want is one that's just friendly to us - then the following is the next issue I see.
If we all think it's so great to be autonomous, to feel like we're doing all of our own work, all of our own thinking, all of our own exploration - then why does anyone want to build an AI in the first place?
Isn't the world, as it is, lacking an all-powerful AI, perfectly suited to our desires of control and autonomy?
Suppose an AI-friendly-to-you exists, and you know that you can always ask it to expand your mind, and download into you everything it knows about the issues you care for, short-circuiting thousands of years of work that it would otherwise take for you to make the same discoveries.
Doesn't it seem pointless to be doing all that work, if you know that FAI can already provide you with all the answers?
Furthermore, again supposing an AI-friendly-to-you exists - you know that you can always ask it to wire-head you. In any given year, there is a negligible, but non-zero probability that you'll succumb to the temptation. Once you do succumb to the temptation, it will feel so great that you will never ever want to do the boring "thinking and doing stuff" again. You will be constantly blissful, and anything you want to know about the universe will be immediately available to you, through a direct interface with FAI.
It doesn't take much to see that, whether it takes a thousand years or a million years before you succumb, you will eventually choose to be wire-headed; you will choose this much sooner than the universe ends; and the vast majority of your total existence will be lived that way.
If the FAI values that we value independence, and values that we value autonomy - which I think it would have to, to be considered properly Friendly - and if wireheading is an threat to our ability to maintain those values, it doesn't make sense that the FAI would make wireheading available for the asking. It makes much more sense that the FAI would actively protect us from wireheading as it would from any other existential threat, in that case.
(Also, just because it would protect us from existential threats, that wouldn't imply that it would protect us fr...
I can conceive of the following 3 main types of meaning we can pursue in life.
1. Exploring existing complexity: the natural complexity of the universe, or complexities that others created for us to explore.
2. Creating new complexity for others and ourselves to explore.
3. Hedonic pleasure: more or less direct stimulation of our pleasure centers, with wire-heading as the ultimate form.
What I'm observing in the various FAI debates is a tendency of people to shy away from wire-heading as something the FAI should do. This reluctance is generally not substantiated or clarified with anything other than "clearly, this isn't what we want". This is not, however, clear to me at all.
The utility we get from exploration and creation is an enjoyable mental process that comes with these activities. Once an FAI can rewire our brains at will, we do not need to perform actual exploration or creation to experience this enjoyment. Instead, the enjoyment we get from exploration and creation becomes just another form of pleasure that can be stimulated directly.
If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them. This enjoyment could be of any type - it could be explorative or creative or hedonic enjoyment as we know it. The most energy efficient way to create any kind of enjoyment, however, is to stimulate the brain-equivalent directly. Therefore, the greatest utility will be achieved by wire-heading. Everything else falls short of that.
What I don't quite understand is why everyone thinks that this would be such a horrible outcome. As far as I can tell, these seem to be cached emotions that are suitable for our world, but not for the world of FAI. In our world, we truly do need to constantly explore and create, or else we will suffer the consequences of not mastering our environment. In a world where FAI exists, there is no longer a point, nor even a possibility, of mastering our environment. The FAI masters our environment for us, and there is no longer a reason to avoid hedonic pleasure. It is no longer a trap.
Since the FAI can sustain us in safety until the universe goes poof, there is no reason for everyone not to experience ultimate enjoyment in the meanwhile. In fact, I can hardly tell this apart from the concept of a Christian Heaven, which appears to be a place where Christians very much want to get.
If you don't want to be "reduced" to an eternal state of bliss, that's tough luck. The alternative would be for the FAI to create an environment for you to play in, consuming precious resources that could sustain more creatures in a permanently blissful state. But don't worry; you won't need to feel bad for long. The FAI can simply modify your preferences so you want an eternally blissful state.
Welcome to Heaven.