Please give an example of why the AGI should co-operate with something that cannot do anything the AGI itself cannot.
If the overall price (including time, gaining requisite knowledge, etc) of co-operation is less expensive than the AGI doing it itself, the AGI should co-operate. No?
If I'm a god, what would I need a human for? If I need humans, I can just make some. Better still, I could replace them with something more efficient that doesn't complain or rebel.
How expensive is making humans vs. their utility? Is there something markedly more efficient that won't complain or rebel if you treat it poorly? How efficient/useful could a human be if you treated it well?
There are also useful pseudo-moral arguments of the type of pre-committing to a benevolent strategy so that others (bigger than you) will be benevolent to you.
The fundamental flaw in your reasoning here is that you keep trying to construct paths through probability space that could support your hypothesis, but ONLY if you had presented some evidence for singling out that hypothesis in the first place!
Agreed. So your argument is that I'm not adequately presenting evidence for singling out that hypothesis. That's a useful criticism. Thanks!
And that's why you're getting so many downvotes: in LW terms, you are failing basic reasoning.
I disagree. I believe that I am failing to successfully communicate my reasoning. I understand your arguments perfectly well (and appreciate them) and agree with them if that is what I was trying to do. Since they are not what I'm trying to do -- although they apparently are what I AM doing -- I'm assuming (yes, ASS-U-ME) that I'm failing elsewhere and am currently placing the blame on my communication skills.
Are you willing to accept that premise and see if you can draw any helpful conclusions or give any helpful advice?
And, once again, thank you for already taking the time to give such a detailed thoughtful response.
How expensive is making humans vs. their utility? Is there something markedly more efficient that won't complain or rebel if you treat it poorly?
Yes. The nano bots that you could build out of my dismantled raw materials.There is something humbling to realise that my complete submission and wholehearted support is worth less to a non-friendly AI than my spleen.
[...] SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.
[...] Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)
So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.
[...] If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.
The line of argument makes sense, if you accept the premises.
But, I don't.
Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It), October 29 2010. Thanks to XiXiDu for the pointer.