= 27d567bc2d48d9f40e9ccc20ea370b15)
orthonormal comments on Decision Theories: A Semi-Formal Analysis, Part III - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (53)
So, let's consider this with three strategies: 1) CooperateBot, 2) DefectBot, 3) CliqueBot.
We now get a table that looks like this (notice the letters are outcomes, not actions):
X has control over which row they get, and Y has control over which column they get. The two Nash equilibria are (2,2) and (3,3). Conveniently, (2,2) and (2,3) have the same result, and so it looks like it'll be easy to figure out that (3,3) is the better Nash equilibrium.
After further review, I was wrong that CDT would be capable of making use of this to act like TDT. If CDT treats its output as separate from the rest of the causal graph (in the sense of the previous post), then it would still prefer to output an always-defect rather than a Löbian mathematical object. So it does take a different kind of agent to think of Nash equilibria among strategies.
Also, the combinatorics of enumerating strategies and looking for Nash equilibria are kind of awful: there are 16 different inputs that such a strategy has to deal with (i.e. what the opponent does against CooperateBot, DefectBot, NewcombBot and AntiNewcombBot), so there are 2^16 variant strategies in the same class. The one we call TDT is the Nash equilibrium, but it would take a long while to establish that in a naive implementation.
When its source code is provided to its opponent, how could it be sensible to treat its output as separate from the rest of the causal graph?
Sure, but it's just as bad for the algorithm you wrote: you attempt to deduce output Y(code G, code A_i) for all A_i, which is exactly what you need to determine the Nash Equilibrium of this table. (Building the table isn't much extra work, if it even requires more, and is done here more for illustrative than computational purposes.)
I am really worried that those two objections were enough to flip your position on this.
No, there are four different A_i (CooperateBot, DefectBot, NewcombBot and AntiNewcombBot). 2^16 is the number of distinct agents one could write that see what Y does against the A_i and picks an action based on those responses. Just taking the maximum each time saves you from enumerating 2^16 strategies.
That is what CDT is. "Sensible" doesn't enter into it.
(To expand on this, CDT's way of avoiding harmful self-reference is to treat its decision as a causally separate node and try out different values for it while changing nothing else on the graph, including things that are copies of its source code. So it considers it legitimate to figure out the impact of its present decision on any agent who can see the effects of the action, but not on any agent who can predict the decision. Don't complain to me, I didn't make this up.)
It's not clear to me that's the case. If your bot and my bot both receive the same source code for Y, we both determine the correct number of potential sub-strategies Y can use, and have to evaluate each of them against each of our A_is. I make the maximization over all of Y's substrategies explicit by storing all of the values I obtain, but in order to get the maximum you also have to calculate all possible values. (I suppose you could get a bit of computational savings by exploiting the structure of the problem, but that may not generalize to arbitrary games.)
The basic decision here is what to write as the source code, not the action that our bot outputs, and so CDT is fine- if it modifies the source code for X, that can impact the outputs of both X and Y. There's no way to modify the output of X without potentially modifying the output of Y in this game and I don't see a reason for CDT to mistakenly hallucinate one.
Put another way, I don't think I would use "causally separate"- I think I would use "unprecedented." The influence diagram I'm drawing for this has three decision boxes (made by three different agents), all unprecedented, whose outputs are the code for X, Y, and G; all of them point to the calculation node of the inference module, and then all three codes and the inference module point to separate calculation nodes of X's output and Y's output, which then both point to the value node of Game Outcome. (You could have uncertainty nodes pointing to X's output and Y's output separately, but I'm ignoring mixed strategies for now.)
To the best of my knowledge, this isn't a feature of CDT: it's a feature of the embedded physics module used by most CDTers. If we altered Newcomb's problem such that Omega filled the boxes after the agent made their choice, then CDTers would one-box- and so if you have a CDTer who believes that perfect prediction is equivalent to information moving backwards in time (and that that's possible), then you have a one-boxing CDTer.