And you don't think a self-improving AI will ever fall into this category? Hell, if you gave a human the ability to run billions of simulations per second to study how their decisions would turn out, they'd be able to take over the world and "remain far, far more powerful" than everyone else. (If they were actually more intelligent, and not just faster, even more so.) Your so-called "limited edge case" is the main case being discussed: superhuman intelligence. (The problem of single-goal entities is of course also discussed here; see the idea of a "paper-clip maximizer", for example.) In short, you seem to be saying that we shouldn't worry about those "edge" cases because in all non-"edge" cases, things work out fine. That's like saying we shouldn't worry about having fire departments or constructing homes according to a fire code, because a fire is an "edge" case, and normally buildings don't burn down. Even if you were to make such an argument, it makes little sense to propose it at a meeting of the fire council. ;-) It may be true that mostly, fires don't happen. However, it's also true that if you don't build the buildings with fire prevention (and especially, preventing the spread of fires) in mind, then, sooner or later, your whole city burns down. Because at that point, it only takes one fire to do it.

I can't accept this. Many animals are not social, or are social only to the extent of practicing parental care. [...] Only if it is actually advantageous to them (it?). Your claim would be much more convincing if you could provide examples of what AIs might gain by social interaction with humans, and why the AI could not achieve the same benefits with less risk and effort by exterminating or enslaving us. Without such examples, your bare assertions are completely unconvincing. Please note that as humans evolved to their current elevated moral plane they occasionally found extermination and enslavement to be more tempting solutions to their social problems than reciprocity. In fact, enslavement is a form of reciprocity - it is one possible solution to a bargaining problem as in Nash(1953). A solution in which one bargainer has access to much better threats than the other.