= 27d567bc2d48d9f40e9ccc20ea370b15)
cousin_it comments on Cooperating with agents with different ideas of fairness, while resisting exploitation - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
Comments (44)
Solution concept implementing this approach (as I understand it):
Player X chooses Pareto fair outcome (X→X, X→Y), (X→Y can be read as "player X's fair utility assignment to player Y"), player Y chooses fair outcome (Y→X, Y→Y).
The actual outcome is (Y→X, X→Y)
(If you have a visual imagination in maths, as I do, you can see this graphically as the Pareto maximum among all the points Pareto worse than both fair outcomes).
This should be unexploitable in some senses, as you're not determining your own outcome, but only that of the other player.
Since it's not Pareto, it's still possible to negotiate over possible improvements ("if I change my idea of fairness towards the middle, will you do it too?") and blackmail is possible in that negotiation process. Interesting idea, though.
If I'm determining the outcome of the other player, doesn't that mean that I can change my "fair point" to threaten the other player with no downside for me? That might also lead to blackmail...
Indeed! And this is especially the case if any sort of negotiations are allowed.
But every system is vulnerable to that. Even the "random dictator", which is the ideal of unexploitability. You can always say "I promise to be a better (worse) dictator if you (unless you) also promise to be better".
If I understand correctly, what Stuart proposes is just a special case of what Eliezer proposes. EY's scheme requires some function mapping the degree of skew in the split to the number of points you're going to take off the total. SA's scheme is the special case where that function is the constant zero.
The more punishing function you use, the stronger incentive you create for others to accept your definition of 'fair', but on the other hand, if the party you're trading with genuinely has a a different concept of 'fair' and if you're both following this technique, it'd be best for both of you to use the more lenient zero-penalty approach.
As far as I can tell, if you've reliably pre-committed to not give in to blackmail (and the other party is supposed to be able to read your source code after all), the zero-penalty approach seems to be optimal.