And yet humanity is resistant to large scale effects because we also combat changes in values that are destructive (like nazism). Are you suggesting that through persuasive means an AI could convert the values of all humanity to something unsavory? I think this is a bit too negative a view on humanity. You might suggest conditioning from birth, but this will result in outrage from the rest of humanity which the AI, by our utility definition, is trying to avoid.
And yet humanity is resistant to large scale effects because we also combat changes in values that are destructive (like nazism).
Ah implicit belief in moral progress (or at least values and morality being preserved) and a universe where really bad things can't happen. I sometimes miss that.
The Nazis where defeated not because they where destructive but because the Soviet Union, UK and the US where stronger. Speaking of Stalin, how does Communism fit into your model?
I put "trivial" in quotes because there are obviously some exceptionally large technical achievements that would still need to occur to get here, but suppose we had an AI with a utilitarian utility function of maximizing subjective human well-being (meaning, well-being is not something as simple as physical sensation of "pleasure" and depends on the mental facts of each person) and let us also assume the AI can model this "well" (lets say at least as well as the best of us can deduce the values of another person for their well-being). Finally, we will also assume that the AI does not possess the ability to manually rewire the human brain to change what a human values. In other words, the ability for the AI to manipulate another person's values is limited by what we as humans are capable of today. Given all this, is there any concern we should have about making this AI; would it succeed in being a friendly AI?
One argument I can imagine for why this fails friendly AI is the AI would wire people up to virtual reality machines. However, I don't think that works very well, because a person (except Cypher from the Matrix) wouldn't appreciate being wired into a virtual reality machine and having their autonomy forcefully removed. This means the action does not succeed in maximizing their well-being.
But I am curious to hear what arguments exist for why such an AI might still fail as a friendly AI.