orthonormal comments on An Alternative Approach to AI Cooperation - Less Wrong

6 Post author: Wei_Dai 31 July 2009 12:14PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (25)

You are viewing a single comment's thread. Show more comments above.

Comment author: Jonathan_Graehl 31 July 2009 06:54:40PM *  0 points [-]

Individual verified-source reincarnation and joint verified-source reincarnation on a single computing substrate are equivalent in every way.

Call A+ the reincarnation of A, and B+ the reincarnation of B.

There must be some way of observing that the irrevocable transfer of authority has actually occurred. This means you need some way of trusting promises by all entities that owe some allegiance to A or B. So far the only method proposed to establish trust in one's decision rule is to submit to verified reincarnation. This may not be to the liking of non-machine intelligence, and at best costly to those involved, if not infeasible due to speed-of-light physical separation.

Further, A+ shouldn't depart, in any way it wasn't willing to do unilaterally for its own sake, from the values and decision rules of A, unless there's a secure atomic transfer of authority from A->A+ and B->B+. I don't see how such a thing is possible, so A+ really can't commit to any cooperation until it verifies that B->B+ has occurred.

So, A+ and B+ would differ from A and B only in that their own end of the contract they propose to enter would become active iff the other actually reincarnates (A+ can do this once B->B+, because she knows that B+ will do the same). But it is necessary for one of A+ or B+ to reincarnate first.

"On the count of 3, hang up!"

Comment author: orthonormal 31 July 2009 07:06:57PM 1 point [-]

The process of transferring resources might be done visibly and gradually, like the iterated donation pact in this comment.

Comment author: Jonathan_Graehl 31 July 2009 11:11:12PM 0 points [-]

I agree that it might be nice to transfer gradually, but if you avoid the risk of losing any fraction, then you solve the problem of one-shot transfer also. If you don't solve that problem, then you still risk losing something to treachery. Maybe you could suggest a concrete protocol for a verified-source reincarnation scenario.

I'm also envisioning some non-divisible resources as well, e.g. privileges and obligations with respect to independent entities (we'll assume these were crafted to transfer on reincarnation).