Discussion for Eliezer Yudkowsky's paper: Timeless Decision Theory

Alexei

I have not seen any place to discuss Eliezer Yudkowsky's new paper, titled Timeless Decision Theory, so I decided to create a discussion post. (Have I missed an already existing post or discussion?)

By "tit for tat" I am referring to the notable strategy in the iterated prisoner's dilemma. Agents using this strategy will keep cooperating as long as the other person cooperates, but if the other person defects then they will defect too. It's an excellent strategy by many measures, beating out more complicated strategies, and we probably have something like it built into our heads.

By analogy, a "tit for tat" strategy in Newcomb's problem with transparent boxes would be to one-box if the Predictor "cooperates," and two-box if the Predictor "defects."

But what does the Predictor see when it looks into the future of an agent with this strategy? Either way it chooses, it will have chosen correctly, so the Predictor needs some other, non-decision-determined criterion to decide.

Alternately you could think of it as making the decision-type of the agent undefined (at the time the Predictor is filling the boxes), thus making it impossible for the problem to have any well-defined decision-determined statement.

Just to clarify, I think your analysis here doesn't apply to the transparent-boxes version that I presented in Good and Real. There, the predictor's task is not necessarily to predict what the agent does for real, but rather to predict what the agent would do in the event that the agent sees $1M in the box. (That is, the predictor simulates what--according to physics--the agent's configuration would do, if presented with the $1M environment; or equivalently, what the agent's 'source code' returns if called with the $1M argument.)

If the agent would one-box ... (read more)

0Vladimir_Nesov16y

This demonstrates that an agent can't know its own decision. In this case, predictor can't know its own prediction, and so can't know agent's action, if that action allows to infer the prediction. (And this limitation can't be fought with computational power, so Omega is as susceptable.) For predictors, it's enough to have a fixpoint, to pick any self-fulfilling prediction. But if the environment is playing diagonal, as you describe, then the predictor can't make a correct prediction. This is not about failure of environment to be decision-determined, the environment you describe simply has the predictor lose for every decision. (If you consider the question in enough detail, the distinction between the decision-determined problems and other kinds of problems doesn't make sense, apart from highlighting that decision can be important apart from action-instance in the environment or other concepts, that all these are different concepts and decision makes sense abstractly, on its own.)