My view is much more engineering than philosophical perspective. Some degree of 'friendliness' (and not in the sense of psychopathic benefactor who will lie and mislead and manipulate you to make your life better, but in the sense of trust) is necessary for intellectual cooperation of AI's nodes. I think your problem is that you define intelligence very narrowly as something which works to directly fitful material needs, goals, etc. and the smarter it is the closer it must be modelled by that ideal (you somewhere lost the concept of winning the most and replaced it with something else). Very impoverished thinking, when combined with ontologically basic 'intelligence' that doesn't really consist of components (with the 'don't build some of the most difficult components' as a solution to problem).
Let baz = SI's meaning of AI. Let koo=industry's meaning of AI. Bazs are incredibly dangerous, and must not be created, i'll take SI's word for it. The baz requires a lot of extra work compared to useful koo (philosophy of mind work that we can't get a koo to do), the work that is clearly at best pointless, at worst dangerous, and definitely difficult; one doesn't need to envision foom and destruction of mankind to avoid implementing extra things that are both hard to make and can only serve to make the software less useful. The only people I know of who want to build a provably friendly baz, is SI. They also call their baz a koo, for sake of soliciting donations for work on baz and for sake of convincing people koo is as dangerous as baz (which they achieve by not caring enough to see the distinction). The SI team acts as a non friendly baz would.
edit: to clarify on the trust friendliness, you don't want a node to model what other nodes would do, that'd duplicated computation; this rules out straightforward utilitarian consequentialism as practically relevant foundational principle because the consequences are not being modelled.
I think your problem is that you define intelligence very narrowly as something which works to directly fitful material needs, goals, etc. and the smarter it is the closer it must be modelled by that ideal (you somewhere lost the concept of winning the most and replaced it with something else).
Defining intelligence as the ability to achieve arbitrary goals is a very narrow definition? What's the broader one?
As Luke mentioned, I am in the process of writing "Responses to Catastrophic AGI Risk": A journal-bound summary of the AI risk problem, and a taxonomy of the societal proposals (e.g. denial of the risk, no action, legal and economic controls, differential technological development) and AI design proposals (e.g. AI confinement, chaining, Oracle AI, FAI) that have been made.
One of the categories is "They Will Need Us" - claims that AI is no big risk, because AI will always have a need of something that humans have, and that they will therefore preserve us. Currently this section is pretty empty:
But I'm certain that I've heard this claim made more often than in just those two sources. Does anyone remember having seen such arguments somewhere else? While "academically reputable" sources (papers, books) are preferred, blog posts and websites are fine as well.
Note that this claim is distinct from the claim that (due to general economic theory) it's more beneficial for the AIs to trade with us than to destroy us. We already have enough citations for that argument, what we're looking for are arguments saying that destroying humans would mean losing something essentially irreplaceable.