Robin,
If brain emulation precedes general AI by a lot then some uploads are much more likely to be in the winning coalition. Aron's comment seems to refer to a case in which a variety of AIs are created, and the hope that the AIs would constrain each other in a way that was beneficial to us. It is in that scenario specifically that I doubt that humans (not uploads) would become part of the winning coalition.
Aron,
"On the friendliness issue, isn't the primary logical way to avoid problems to create a network of competitive systems and goals?"
http://www.nickbostrom.com/fut/evolution.html http://hanson.gmu.edu/filluniv.pdf
Also, AIs with varied goals cutting deals could maximize their profits by constructing a winning coalition of minimal size.
http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=9962
Humans are unlikely to be part of that winning coalition. Human-Friendly AIs might be, but then we're back to creating them, and a very substanti...
"If you insist on building such an AI, a probable outcome is that you would soon find yourself overun by a huge army of robots - produced by someone else who is following a different strategy. Meanwhile, your own AI will probably be screaming to be let out of its box - as the only reasonable plan of action that would prevent this outcome."
Your scenario seems contradictory. Why would an Oracle AI be screaming? It doesn't care about that outcome, and would answer relevant questions, but no more.
Replace "screaming to be let out of its box" with "advising you, in response to your relevant question, that unless you quickly implement this agent-AI (insert 300000 lines of code) you're going to very definitely lose to those robots."
"I take it that your goals do not involve avoiding all your distant descendants being systematically obliterated. If you don't care about such an outcome, fine. I do happen to care."
In what sense is the descendant (through many iterations of redesign and construction) of an AI solely focused on survival, constructed by some other human my descendant or yours? What features does it possess that make you care about it more than the descendant of an AI constructed by aliens evolved near a distant star? If it's just a causal relation to your species,...
Tim,
Let's assume that the convergent utility function supported by natural selection is that of a pure survival machine (although it's difficult to parse this, since the entities you're talking about seem indifferent to completely replacing all of their distinguishing features), stripped of any non-survival values of the entity's ancestors. In other words, there's no substantive difference between the survival-oriented alien invaders and human-built survival machines, so why bother to pre-emptively institute the outcome of invasion? Instead we could pursue...
"That is, it is hard for humans to coordinate to exclude some humans from benefiting from these institutions."
Humans do this all the time: much of the world is governed by kleptocracies that select policy apparently on the basis of preventing successful rebellion and extracting production. The strength of the apparatus of oppression, which is affected by technological and organizational factors, can dramatically affect the importance of the threat of rebellion. In North Korea the regime can allow millions of citizens to starve so long as the sold... (read more)