The AI is not supposed to change it values, regardless of whether it is powerful enough to realize them. Values are not up for grabs. Once the AI has some values it either wins and reshapes reality according to them or loses.
A remarkably strong claim.
My initial reaction is that humanity's values have certainly changed over time. I think it would require some rather unattractive mental gymnastics to claim that people who beat their children for their own good and people who owned slaves and people who beat, killed, and/or raped either slaves or other people they had vanquished as their right "really" had the same values we currently have, but just hadn't really thought them through, or that our values applied in their world would have lead us to similar beliefs about right and wrong.
I had even thought my own values had changed over my lifetime. I'm not as sure of that, but what about that?
Certainly, it seems, as the human species has evolved its values have changed. Do chimpanzees and bonobos have different values than we do, or the same? If the same, I'd love to see your mental gymnastics to justify that, I would expect them to be ugly. If different, does this mean that our common ancestor has necessarily "lost," assuming its values were some intermediate between ours, chimps, and bonobos, and all of its descendants have different values than it had?
As I understand the word values, our values have changed over time, different groups of humans have some different values from each other, and if there is a "kernel" of common values in our species, that this kernel most likely differs from the kernel of values in homo neanderthalis or other sentient predecessors of modern homo sapiens.
So if NI (Natural Intelligence) in its evolution can change values (can it?) with generally broad consensus that "we" have not lost in this process, why would an AI be precluded from futzing with its values as it worked on self-modifying to increase its intelligence?
Because, if the AI worked, it would consider the fact that if it changed its values, they would be less likely to be maximised, and would therefore choose not to change its values. If the AI wants the future to be X, changing itself so that it wants the future to be Y is a poor strategy for achieving its aims - the future will end up not-X if it does that. Yes, humans are different. We're not perfectly rational. We don't have full access to our own values to begin with, and if we did we might sometimes screw up badly enough that our values change. An FAI ought to be better at this stuff than we are.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.