= 27d567bc2d48d9f40e9ccc20ea370b15)
orthonormal comments on Decision Theories: A Semi-Formal Analysis, Part III - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (53)
Well, it certainly will defect against any mixed strategy that is hard coded into the opponent’s source code. On the other hand, if the mixed strategy the opponent plays is dependent on what it predicts the TDT agent will play, then the TDT agent will figure out which outcome has a higher expected utility:
(I defect, Opponent runs "defection predicted" mixed strategy)
(I cooperate, Opponent runs "cooperation detected" mixed strategy)
Of course, this is still simplifying things a bit, since it assumes that the opponent can perfectly predict one's strategy, and it also rules out the possibility of the TDT agent using a mixed strategy himself.
Thus the actual computation is more like
ArgMax(Sum(ExpectedUtility(S,T)*P(T|S)))
where the argmax is over S: all possible mixed strategies for the TDT agent
the sum is over T: all possible mixed strategies for the opponent
and P(T|S) is the probability that opponent will play T, given that we choose to play S. (so this is essentially an estimate of the opponent's predictive power.)
Won't that let the opponent steal utility from you? Consider the case where you're going up against another TDTer which is willing to consider both the strategy "if they cooperate only if I cooperate, then cooperate with 99% probability" and "if they cooperate only if I cooperate, then cooperate." You want your response to the first strategy to be defection and your response to the second strategy to be cooperation, so it's in their interests to play the second strategy.
You've made it into a bargaining game with that mixed strategy, and indeed the version of TDT we introduced will defect against an opponent that outputs the mixed strategy (if that opponent would output a pure cooperate if that were the only way to get its adversary to cooperate). But bargaining games are complicated, and I'm saving that for another post.