= 27d567bc2d48d9f40e9ccc20ea370b15)
dxu comments on The True Prisoner's Dilemma - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (112)
I think what might be confusing is that your decision depends on what you know about the paperclip maximizer. When I imagine myself in this situation, I imagine wanting to say that I know "nothing". The trick is, if you want to go a step more formal than going with your gut, you have to say what your model of knowing "nothing" is here.
If you know (with high enough probability), for instance, that there is no constraint either causal or logical between your decision and Clippy's, and that you will not play an iterated game, and that there are no secondary effects, then I think D is indeed the correct choice.
If you know that you and Clippy are both well-modeled by instances of "rational agents of type X" who have a logical constraint between your decisions so that you will both decide the same thing (with high enough probability), then C is the correct choice. You might have strong reasons to think that almost all agents capable of paperclip maximizing at the level of Clippy fall into this group, so that you choose C.
(And more options than those two.)
The way I'd model knowing nothing in the scenario in my head would be something like the first option, so I'd choose D, but maybe there's other information you can get that suggests that Clippy will mirror you, so that you should choose C.
It does seem like implied folk-lore that "rational agents cooperate", and it certainly seems true for humans in most circumstances, or formally in some circumstances where you have knowledge about the other agent. But I don't think it should be true in principal that "optimization processes of high power will, with high probability, mirror decisions in the one-shot prisoner's dilemma"; I imagine you'd have to put a lot more conditions on it. I'd be very interested to know otherwise.
I understood that Clippy is a rational agent, just one with a different utility function. The payoff matrix as described is the classic Prisoner's dilemma where one billion lives is one human utilon and one paperclip on Clippy utilon; since we're both trying to maximise utilons, and we're supposedly both good at this we should settle for (C,C) over (D,D).
Another way of viewing this would be that my preferences run thus: (D,C);(C,C);(D,D);(C,D) and Clippy run like this: (C,D);(C,C);(D,D);(D,C). This should make it clear that no matter what assumptions we make about Clippy, it is universally better to co-operate than defect. The two asymmetrical outputs can be eliminated on the grounds of being impossible if we're both rational, and then defecting no longer makes any sense.
Wait, what? You prefer (C,D) to (D,D)? As in, you prefer the outcome in which you cooperate and Clippy defects to the one in which you both defect? That doesn't sound right.
woops, yes that was rather stupid of me. Should be fixed now, my most preferred is me backstabbing Clippy, my least preferred is him backstabbing me. In the middle I prefer cooperation to defection. That doesn't change my point that since we both have that preference list (with the asymmetrical ones reversed) then it's impossible to get either asymmetrical option and hence (C,C) and (D,D) are the only options remaining. Hence you should co-operate if you are faced with a truly rational opponent.
I'm not sure whether this holds if your opponent is very rational, but not completely. Or if that notion actually makes sense.