"The popular dystopian vision of AI is wrong for one simple reason: it equates intelligence with autonomy. That is, it assumes a smart computer will create its own goals, and have its own will (...)
It must be motivated cognition. I refuse to believe a smart person can come to this conclusion without a blindspot the size of, say, the fate of humanity.
A general problem solving agent is just one sloppy "problem to solve is 'make the trains run on time'" away from trying to exterminate the human race (to make the trains run on time, d'uh). A 'tool AI' is just one "do { } while (condition)" loop away from being an agent. These variants are all trivially transformed into one another, once you have the code for a general problem solver.
Now, that shouldn't be too hard to grok. Unless your conscience depends on not grokking it, I suppose.
So should intelligence and autonomy be equated? If not, what is their relationship?
I think this is a very hard problem, and autonomy/agency might actually be even harder to measure/predict/create than intelligence. The question of free will, already really difficult, is just a special case of the question of what constitutes an agent. And to solve it we'd have to get past at least four problems that the study of intelligence doesn't have (anymore, to the same degree):
https://medium.com/backchannel/ai-wont-exterminate-us-it-will-empower-us-5b7224735bf3
(Slashdot discussion: http://tech.slashdot.org/story/14/12/10/1719232/ai-expert-ai-wont-exterminate-us----it-will-empower-us)
Not sure what the local view of Oren Etzioni or the Allen Institute for AI is, but I'm curious what people think if his views on UFAI risk. As far as I can tell from this article, it basically boils down to "AGI won't happen, at least not any time soon." Is there (significant) reason to believe he's wrong, or is it simply too great a risk to leave to chance?