I don't follow and understand the "timeless decision" topic on LW, but I have a feeling that a significant part of that is one agent predicting what other agent would do, by simulating their algorithm. (This is my very uninformed understanding of the "timeless" part: I don't have to wait until you do X, because I can already predict if you would do X, and behave accordingly. And you don't have to wait for my reaction, because you can already predict it too. So let's predict-cause each other to cooperate, and win mutually.)
If I am correct, there is a problem with this: having an access to another agent's code does not allow you to make any conclusions, in general case.
You can only make a simulation of one specific situation. Then another. Hoping that the agent does not want to run your simulation, which would get you both into an infinite loop. And you can't even tell whether the agent wants to run your simulation, or not.
I don't follow and understand the "timeless decision" topic on LW, but I have a feeling that a significant part of that is one agent predicting what other agent would do, by simulating their algorithm.
Thinking in terms of "simulating their algorithm" is convenient for us because we can imagine the agent doing it and for certain problems a simulation is sufficient. However the actual process involved is any reasoning at all based on the algorithm. That includes simulations but also includes creating mathematical proofs based on the al...
If it's worth saying, but not worth its own post, even in Discussion, it goes here.