The decorator follows the implied wishes not because he is smart enough to know what they are, but because he wishes to act in his client's interest to gain payment, reputation, etc.
And an AGI wishes to achieve its goals the way they are meant to be achieved. Which includes all implicit conditions.
An AGI does not have to explicitly care about humans and their values as long as the implied context of its goals is human volition.
Consider a rich but sociopathic human decorator who solely cares about being a good decorator. What does a good decorator do? It does what its contract explicitly tells him to do AND what is implied by it, including the satisfaction of the customer.
You don't need human moral values or any other complex values as long as you care to achieve your goals the way they are meant to be achieved, explicitly and implicitly.
If an AI has human interests as its main goal, it is already friendly. The question was whether intelligence on its own is enough to align it with human interests, which seems very unlikely. If the AI actually has cooperation with humans or fulfillment of some human wish as its goal, it will be able to use intelligence to better fulfill the wishes with all available context. But it's getting the AI to operate with that goal that is difficult, I believe.
One of the most annoying arguments when discussing AI is the perennial "But if the AI is so smart, why won't it figure out the right thing to do anyway?" It's often the ultimate curiosity stopper.
Nick Bostrom has defined the "Orthogonality thesis" as the principle that motivation and intelligence are essentially unrelated: superintelligences can have nearly any type of motivation (at least, nearly any utility function-bases motivation). We're trying to get some rigorous papers out so that when that question comes up, we can point people to standard, and published, arguments. Nick has had a paper accepted that points out the orthogonality thesis is compatible with a lot of philosophical positions that would seem to contradict it.
I'm hoping to complement this with a paper laying out the positive arguments in favour of the thesis. So I'm asking you for your strongest arguments for (or against) the orthogonality thesis. Think of trying to convince a conservative philosopher who's caught a bad case of moral realism - what would you say to them?
Many thanks! Karma and acknowledgements will shower on the best suggestions, and many puppies will be happy.