I'll just comment on what most people are missing, because most reactions seem to be missing a similar thing.

Wei explains that most of the readership are preference utilitarians, who believe in satisfying people's preferences, not maximizing pleasure.

That's fine enough, but if you think that we should take into account the preferences of creatures that could exist, then I find it hard to imagine that a creature would prefer not to exist, than to exist in a state where it permanently experiences amazing pleasure.

Given that potential creatures outnumber existing creatures many times over, the preferences of existing creatures - that we wish to selfishly keep the universe's resources to ourselves, so we can explore and think and have misguided lofty impressions about ourselves, and whatnot - all of those preferences don't count that much in the face of many more creatures that would prefer to exist, and be wireheaded, than not to exist at all.

The only way preference utilitarianism can avoid the global maximum of Heaven is to ignore the preferences of potential creatures. But that is selfish.

If you don't want Heaven, then you don't want a universally friendly AI. What you really want is an AI that is friendly just to you.