assume the AGI will have the capacity to kill us at very low cost/risk
This assumption comes at a high cost in probability mass. The difficulty of "killing off humanity" type tasks will increase exponentially as AI leads to AGI leads to super-AGI; its a moving target.
humans make very inefficient use of resources (including what we need to stay alive)
Largely irrelevant: humans use an infinitesimal fraction of solar resources. Moreover, (bacteria, insects, rats) make very inefficient use of our resources as well, why haven't we killed them off?
most AGI goals are improved by more resources
The bronze age did not end for lack of steam, nor the coal age for lack of coal. Evolution appears to move forward by using less resources rather than more.
most AGI goals are not human friendly
Who cares? Most AGI goals will never be realized.
Hence most AGIs will make better use of resources by controlling them than by trading with humans.
True, the question is: what resources?
Hence most AGIs will kill us by default.
Most random home brain surgical operations will kill us by default as well.
One of the most annoying arguments when discussing AI is the perennial "But if the AI is so smart, why won't it figure out the right thing to do anyway?" It's often the ultimate curiosity stopper.
Nick Bostrom has defined the "Orthogonality thesis" as the principle that motivation and intelligence are essentially unrelated: superintelligences can have nearly any type of motivation (at least, nearly any utility function-bases motivation). We're trying to get some rigorous papers out so that when that question comes up, we can point people to standard, and published, arguments. Nick has had a paper accepted that points out the orthogonality thesis is compatible with a lot of philosophical positions that would seem to contradict it.
I'm hoping to complement this with a paper laying out the positive arguments in favour of the thesis. So I'm asking you for your strongest arguments for (or against) the orthogonality thesis. Think of trying to convince a conservative philosopher who's caught a bad case of moral realism - what would you say to them?
Many thanks! Karma and acknowledgements will shower on the best suggestions, and many puppies will be happy.