The popular dystopian vision of AI is wrong for one simple reason: it equates intelligence with autonomy. That is, it assumes a smart computer will create its own goals, and have its own will (...)
The distinction is formally correct. But I agree that autonomy comes in very quickly by attaching a read-eval-print loop around the optimizer which takes the state of the world as input for the maximzation.
It's not even formally correct. An autonomous AI does not need to create its own terminal goals*, and the will we give it is perfectly adequate to screw us over.
https://medium.com/backchannel/ai-wont-exterminate-us-it-will-empower-us-5b7224735bf3
(Slashdot discussion: http://tech.slashdot.org/story/14/12/10/1719232/ai-expert-ai-wont-exterminate-us----it-will-empower-us)
Not sure what the local view of Oren Etzioni or the Allen Institute for AI is, but I'm curious what people think if his views on UFAI risk. As far as I can tell from this article, it basically boils down to "AGI won't happen, at least not any time soon." Is there (significant) reason to believe he's wrong, or is it simply too great a risk to leave to chance?