Another take at clarifying that UDT seems to say, to add to the discussion at the end of the post: there is no way to change not only the past, the future (see: free will), but also counterfactuals. What was, and what will be is fixed to what it actually was and what it actually will be, but the same applies to what could be, that is what could be is also fixed to what it actually could be. This doesn't interfere with the ability to determine what will be, what was, and what could be. And that's all UDT says: one can't temper with what one could've done, in the sense of changing it, although one can determine it.
To bind preference, shouldness to this factual setting, it's better to forget the details of expected utility maximization algorithm, and say that what the agent thinks it should do (at the moment, which may be a very limited perspective that doesn't reflect all of the agent's preference, but only that part that acts at that moment) is what it actually does. Thus, different preferences simply correspond to different algorithms for choosing actions, or more generally to different ways in which the agent determines the (dependence between) past, future, and counterfactuals.
Now if we get back to the point that all these things are "fixed" and are "as they actually are", we can see that there can be no rational disagreement, about anything, ever. One can't disagree about facts, but one can't also disagree about values, when values are seen as a counterpart of actions, facts also. Of course, different agents are different systems, and so they get located by different observations and perform different actions, and in this sense can be said to have different states on knowledge and act on different values, but this is a fact about dots on the picture, not about the picture whole.
(Of course, counterfactuals are the only real thing in the context of this discussion, "past" and "future" aren't concepts appropriately homogeneous here, when I say "determine the future", I mean "determine the 'counterfactuals' that branch in the future".)
What was, and what will be is fixed to what it actually was and what it actually will be, but the same applies to what could be, that is what could be is also fixed to what it actually could be.
As it could have been in the beginning, so it could have been now, and forever could have been going to be.
This continues my previous post on Robin Hanson's pre-rationality, by offering some additional comments on the idea.
The reason I re-read Robin's paper recently was to see if it answers a question that's related to another of my recent posts: why do we human beings have the priors that we do? Part of that question is why are our priors pretty close to each other, even if they're not exactly equal. (Technically we don't have priors because we're not Bayesians, but we can be approximated as Bayesians, and those Bayesians have priors.) If we were created by a rational creator, then we would have pre-rational priors. (Which, since we don't actually have pre-rational priors, seems to be a good argument against us having been created by a rational creator. I wonder what Aumann would say about this?) But we have other grounds for believing that we were instead created by evolution, which is not a rational process, in which case the concept doesn't help to answer the question, as far as I can see. (Robin never claimed that it would, of course.)
The next question I want to consider is a normative one: is pre-rationality rational? Pre-rationality says that we should reason as if we were pre-agents who learned about our prior assignments as information, instead of just taking those priors as given. But then, shouldn't we also act as if we were pre-agents who learned about our utility function assignments as information, instead of taking them as given? In that case, we're led to the conclusion that we should all have common utility functions, or at least that pre-rational agents should have values that are much less idiosyncratic than ours. This seems to be a reductio ad absurdum of pre-rationality, unless there is an argument why we should apply the concept of pre-rationality only to our priors, and not to our utility functions. Or is anyone tempted to bite this bullet and claim that we should apply pre-rationality to our utility functions as well? (Note that if we were created by a rational creator, then we would have common utility functions.)
The last question I want to address is one that I already raised in my previous post. Assuming that we do want to be pre-rational, how do we move from our current non-pre-rational state to a pre-rational one? This is somewhat similar to the question of how do we move from our current non-rational (according to ordinary rationality) state to a rational one. Expected utility theory says that we should act as if we are maximizing expected utility, but it doesn't say what we should do if we find ourselves lacking a prior and a utility function (i.e., if our actual preferences cannot be represented as maximizing expected utility).
The fact that we don't have good answers for these questions perhaps shouldn't be considered fatal to pre-rationality and rationality, but it's troubling that little attention has been paid to them, relative to defining pre-rationality and rationality. (Why are rationality researchers more interested in knowing what rationality is, and less interested in knowing how to be rational? Also, BTW, why are there so few rationality researchers? Why aren't there hordes of people interested in these issues?)
As I mentioned in the previous post, I have an idea here, which is to apply some concepts related to UDT, in particular Nesov's trading across possible worlds idea. As I see it now, pre-rationality is mostly about the (alleged) irrationality of disagreements between counterfactual versions of the same agent, when those disagreements are caused by irrelevant historical accidents such as the random assortment of genes. But how can such agents reach an agreement regarding what their beliefs should be, when they can't communicate with each other and coordinate physically? Well, at least in some cases, they may be able to coordinate logically. In my example of an AI whose prior was picked by the flip of a coin, the two counterfactual versions of the AI are similar enough to each other and symmetrical enough, for each to infer that if it were to change its prior from O or P to Q, where Q(A=heads)=0.5, the other AI would do the same, but this inference wouldn't be true for any Q' != Q, due to lack of symmetry.
Of course, in the actual UDT, such "changes of prior" do not literally occur, because coordination and cooperation between possible worlds happen naturally as part of deciding acts and strategies, while one's preferences stay constant. Is that sufficient, or do we really need to change our preferences and make them pre-rational? I'm not sure.