# Wei_Dai comments on Freaky Fairness - Less Wrong

9 25 July 2009 08:14AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

Comment author: 27 July 2009 08:28:46AM 0 points [-]

The general case of incomplete information also looks bewilderingly hard; I don't even understand what "equilibrium" means in this context.

The Bayesian Equilibrium is usually used in non-cooperative games with incomplete information. (I'm not sure if that's what you're asking.)

BTW, I think another way to formulate this problem, which should be equivalent but might be easier to think about, is instead of the AIs knowing each other's source code and detecting friends and enemies, all the AIs just agree to merge into a single AI running the fair decision algorithm. I had some very preliminary ideas on this which I wrote about on SL4, which I now recognize as trying to reinvent parts of cooperative game theory.

Also very curious about Nesov's approach, but he has told me mutiple times that there's nothing to write up mathematically yet.

What he wrote before led me to look into cooperative game theory, so I'm very curious about his current approach, even if the results aren't ready yet.

Comment author: 27 July 2009 10:12:03AM *  0 points [-]

Ah, you mean this. Okay.

If you're thinking about alternate formulations, it might interest you that programs that know each other's source code can use the quining trick to emulate any mutually-agreed rule for resolving conflicts, e.g. "play a game of chess to decide who gets the money". (Quining ensures that both players' internal representations of other players move the chess pieces correctly.) More generally, they can agree on any common algorithm that takes input strings supplied by individual players (e.g. chess-playing strategies). I guess this counts as "merging"? Another application is generating common cryptographically secure random numbers. This stuff is almost too powerful.

Comment author: 27 July 2009 10:55:06AM *  0 points [-]

The set of things you can do with "knowing each other's source code" seems to be the same as what you can do with "agreeing to merge together". And the underlying mechanisms necessary for implementing them seem to be the same as well.

In order for two AIs meeting for the first time to prove their source code to each other, they each need to reconstruct itself from scratch while being monitored by the other. Similarly, merging is jointly constructing a new entity and then transferring all resources to it. In both cases the necessary enabling technology might be called secure joint construction. (I just made that phrase up. Is there's a better term?)

Comment author: 27 July 2009 11:31:20AM *  1 point [-]

It would help if you gave an example of two simple programs that undergo "secure joint construction" in some formal setting and without any source-code equality tricks. For example, assume that initially they both implement algorithms that want to agree but this fact is Rice-unknowable. This stuff isn't obvious to me. It would be progress.

Comment author: 27 July 2009 11:38:27AM 0 points [-]

Well, AI 1 sends a proposal for a joint decision algorithm to AI 2, then AI 2 sends a counter-proposal and they bargain. They do this until they agree on the joint decision algorithm. They then jointly build a new AI and each monitors the construction process to ensure that the new AI really implements the joint decision algorithm that they agreed on. Finally they each transfer all resources to the new AI and shut down.