cousin_it comments on AI cooperation in practice - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (157)
Edit: The following is not obviously possible, see this comment.
This can be generalized to provide even better algorithms for more general games. Let the following be a simplified outline of the algorithm for cooperation problem, with bounds omitted (players are A and B, our player is A):
The proof that the outcome is "Cooperate" basically hinges on the provability of Löb statement
from which we conclude that A()==B() and so A()=="Cooperate". The statement can be constructed from
and so
Now, we can get basically the same proof if we use a simpler algorithm for the players:
This gets rid of the more general relation, but achieves basically the same result. We can improve on this by defecting against cooperating rocks:
This suggests a general scheme for construction of decision-making algorithms: for all possible actions a and b of players A and B, sort the pairs <a,b> by utility for A, starting from highest, and implement the Löb algorithm for each possibility in turn:
This plays against arbitrary opponents quite well.
According to my experience, the study of game-playing programs has two independent components: fairness/bargaining and enforcement. If you have a good method of enforcement (quining, löbbing, Wei Dai's joint construction), you can implement almost any concept of fairness. Even though Freaky Fairness relied on quining, I feel you can port it to löbbing easily enough. But fairness in games with non-transferable utility is so difficult that it should really be studied separately, without thinking of a specific enforcement mechanism.
Bargaining is about logical uncertainty, something we avoid using sufficiently high upper bounds in this post.