an AI would be human-foolish to take the stupid short-sighted shortcut of trashing us for no reason
You don't seem to understand how basic reasoning works (by LW standards). AFAICT, you are both privileging your hypothesis, and not weighing any evidence.
(Heck, you're not even stating any evidence, only relying on repeated assertion of your framing of the situation.)
You still haven't responded, for example, to my previous point about human-bacterium empathy. We don't have empathy for bacteria, in part because we see them as interchangeable and easily replaced. If for some reason we want some more E. coli, we can just culture some.
In the same way, a superhuman intelligence that anticipates a possible future use for human beings, could always just keep our DNA on file... with a modification or two to make us more pliable.
Your entire argument is based on an enormous blind spot from your genetic heritage: you think an AI would inherently see you as, well, "human", when out of the space of all possible minds, the odds of a given AI seeing you as worth bothering with are negligible at best. You simply don't see this, because your built-in machinery for imagining minds automatically imagines human minds -- even when you try to make it not do so.
Hell, the human-bacterium analogy is a perfect example: I'm using that example specifically because it's a human way of thinking, even though it's unlikely to match the utter lack of caring with which an arbitrary AGI is likely to view human beings. It's wrong to even think of it as "viewing", because that supposes a human model.
AI's are not humans, unless they're built to be humans, and the odds of them being human by accident are negligible.
Remember: evolution is happy to have elephants slowly starve to death when they get old, and to have animals that die struggling and painfully in the act of mating. Arbitrary optimization processes do not have human values.
Stop thinking "intellect" (i.e. human) and start thinking "mechanical optimization process".
[edit to add: "privileging", which somehow got eaten while writing the original comment]
you are both your hypothesis
you are both privileging your hypothesis ?
[...] SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.
[...] Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)
So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.
[...] If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.
The line of argument makes sense, if you accept the premises.
But, I don't.
Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It), October 29 2010. Thanks to XiXiDu for the pointer.