Like any educated denizen of the 21st century, you may have heard of World War II. You may remember that Hitler and the Nazis planned to carry forward a romanticized process of evolution, to breed a new master race, supermen, stronger and smarter than anything that had existed before.
Actually this is a common misconception. Hitler believed that the Aryan superman had previously existed—the Nordic stereotype, the blond blue-eyed beast of prey—but had been polluted by mingling with impure races. There had been a racial Fall from Grace.
It says something about the degree to which the concept of progress permeates Western civilization, that the one is told about Nazi eugenics and hears "They tried to breed a superhuman." You, dear reader—if you failed hard enough to endorse coercive eugenics, you would try to create a superhuman. Because you locate your ideals in your future, not in your past. Because you are creative. The thought of breeding back to some Nordic archetype from a thousand years earlier would not even occur to you as a possibility—what, just the Vikings? That's all? If you failed hard enough to kill, you would damn well try to reach heights never before reached, or what a waste it would all be, eh? Well, that's one reason you're not a Nazi, dear reader.
It says something about how difficult it is for the relatively healthy to envision themselves in the shoes of the relatively sick, that we are told of the Nazis, and distort the tale to make them defective transhumanists.
It's the Communists who were the defective transhumanists. "New Soviet Man" and all that. The Nazis were quite definitely the bioconservatives of the tale.
The whole point of attempting a "Friendly AI" is that its proponents believe that it IS possible to exclude entire branches of possibility from an AI's courses of action - that the superhuman intelligence can be made safe. Not merely friendly in a human sense, but favorable to human interests, not 'evil'.
Of course, they cannot provide an objective and rigorous description of what "being in human interests" actually entails, nor can they explain clearly what 'evil' is. But they know it when they see it, apparently. And since many of them seem to believe that 'values' are arbitrary, they've never bothered subjecting what they value to analysis.
Perhaps the possibility that a consequence of an entity being utterly good might be its being utterly unsafe has never occurred to them. And perhaps the possibility that superhuman general intelligence might analyze their values and find them lacking never occurred to them either. That would explain a lot.
Why would being good make you unsafe?