The fear focuses on the effects of artificial superintelligence, not the effects of artificial intelligence; but it is anticipated that artificial intelligence leads easily to artificial superintelligence, when AI itself is applied to the task of AI (re)design. If you think of an AGI's capabilities as vaguely like the capabilities of a human being, then the appearance of an AI in the world is just like adding one person to a world that already contains 7 billion persons. It might be a historic development, but not an apocalyptic one. And that is indeed how it should turn out, for a large class of possible AIs.
But in a world with AIs, eventually you will have someone or something go down a path that leads, whether by accident or by design, to AI, AI networks, or human-AI networks, that are effectively working to take over the world. A computer virus is a primitive example of software that runs as wild as it can within its environment. There was no law of nature which protected us from having to deal with a world of computer viruses, and there can't be any law of nature which means we'll never have to deal with would-be hegemonic AIs, because trying to take over the world is already cognitively possible for mere humans.
So, if you're going to concern yourself with this possibility at all, either you try to prevent such AI from ever coming into being, or you try to design a benevolent AI which would still be benevolent even if it became all-powerful. Obviously, the Singularity Institute is focused mostly on the second option.
In your comment you talk about safety, so I assume you agree there is some sort of "AI danger", you just think SI has lots of the details wrong. My opinion is, they have certain basics right, but these basics are buried in the discourse by transhumanist hyperbole about the future, by various extreme thought-experiments, by metaphysical hypotheses which have assumed an unwarranted centrality in discussion, and by posturing and tail-chasing to do with "rationality".
The fear focuses on the effects of artificial superintelligence, not the effects of artificial intelligence; but it is anticipated that artificial intelligence leads easily to artificial superintelligence, when AI itself is applied to the task of AI (re)design.
Well, given enough computing power, AIXI-tl is an artificial superintelligence. It also doesn't relate abstract mathematical self and the substrate that approximately computes it's abstract mathematical self; it can't care about the survival of the physical system that approximately computes it; i...
As Luke mentioned, I am in the process of writing "Responses to Catastrophic AGI Risk": A journal-bound summary of the AI risk problem, and a taxonomy of the societal proposals (e.g. denial of the risk, no action, legal and economic controls, differential technological development) and AI design proposals (e.g. AI confinement, chaining, Oracle AI, FAI) that have been made.
One of the categories is "They Will Need Us" - claims that AI is no big risk, because AI will always have a need of something that humans have, and that they will therefore preserve us. Currently this section is pretty empty:
But I'm certain that I've heard this claim made more often than in just those two sources. Does anyone remember having seen such arguments somewhere else? While "academically reputable" sources (papers, books) are preferred, blog posts and websites are fine as well.
Note that this claim is distinct from the claim that (due to general economic theory) it's more beneficial for the AIs to trade with us than to destroy us. We already have enough citations for that argument, what we're looking for are arguments saying that destroying humans would mean losing something essentially irreplaceable.