You make my point right there. World War 2. We went to war in defiance of nazis and refused to be assimilated. People in Germany didn't even like what the nazis were doing. And finally, the nazis didn't care about our outrage and death in the resulting war. An AI trying to maximize well-being, will care profoundly about that, by definition.
The point that other humans fought against it doesn't change the central point that a very large fraction of humans could have a radically different effective morality. Moreover, if Germany hadn't gone to war but had instead done the exact same thing to its internal minorities, most of the world likely would not have intervened.
If you don't like this example so much, one can just look at changing attitudes on many issues. See for example Pinker's book "The Better Angels of Our Nature" where he documents extreme changes in historical attitudes abo...
I put "trivial" in quotes because there are obviously some exceptionally large technical achievements that would still need to occur to get here, but suppose we had an AI with a utilitarian utility function of maximizing subjective human well-being (meaning, well-being is not something as simple as physical sensation of "pleasure" and depends on the mental facts of each person) and let us also assume the AI can model this "well" (lets say at least as well as the best of us can deduce the values of another person for their well-being). Finally, we will also assume that the AI does not possess the ability to manually rewire the human brain to change what a human values. In other words, the ability for the AI to manipulate another person's values is limited by what we as humans are capable of today. Given all this, is there any concern we should have about making this AI; would it succeed in being a friendly AI?
One argument I can imagine for why this fails friendly AI is the AI would wire people up to virtual reality machines. However, I don't think that works very well, because a person (except Cypher from the Matrix) wouldn't appreciate being wired into a virtual reality machine and having their autonomy forcefully removed. This means the action does not succeed in maximizing their well-being.
But I am curious to hear what arguments exist for why such an AI might still fail as a friendly AI.