ESRogs comments on Cooperating with agents with different ideas of fairness, while resisting exploitation - Less Wrong

38 Post author: Eliezer_Yudkowsky 16 September 2013 08:27AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (44)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 16 September 2013 12:46:31PM 2 points [-]

If I'm determining the outcome of the other player, doesn't that mean that I can change my "fair point" to threaten the other player with no downside for me? That might also lead to blackmail...

Comment author: ESRogs 17 September 2013 03:55:44AM 1 point [-]

If I understand correctly, what Stuart proposes is just a special case of what Eliezer proposes. EY's scheme requires some function mapping the degree of skew in the split to the number of points you're going to take off the total. SA's scheme is the special case where that function is the constant zero.

The more punishing function you use, the stronger incentive you create for others to accept your definition of 'fair', but on the other hand, if the party you're trading with genuinely has a a different concept of 'fair' and if you're both following this technique, it'd be best for both of you to use the more lenient zero-penalty approach.

As far as I can tell, if you've reliably pre-committed to not give in to blackmail (and the other party is supposed to be able to read your source code after all), the zero-penalty approach seems to be optimal.