I disagree. UDT does not give an answer to this.
The distinction between 2 and 3 is still there. 2 is still the strategy that chooses the action (not the function from inputs to actions) that Bob would like you to have chosen. 3 chooses the function from inputs to actions which 3 would like you to have chosen. Those two options are different no matter what decision theory you use.
From your claim that UDT gives 2, I think you did not understand what I meant by option 2. I think that what you mean when you say option 2 with UDT is what I mean when I say option 3. Yes, UDT doesnt update, but what option 3 is supposed to mean is that you choose the option you think Bob would want if he knew what you do. i.e. choose the function from input to output that bob would want. Option 2 was really meant to ignore your input and just care about what Bob would want you to choose not given your input. Again, I trust that this is just bad communication, and with my definitions of the 3 options you meant to say UDT says 3. Let me know if you think I am wrong about your intention.
I also disagree with the claim that UDT says 3 (or 2) is better than 1. The question is about should we take Bobs utility as a term in our own utility function (and then multiply it by our probabilities), or should we take Bob's expected utility of our action as a term in our own utility function (which already has his probability built into it). This is a question about how we think that we should allow other peoples utility function to influence our own. UDT doesn't tell us what kind of utility functions we should have.
Again, I trust that this is just bad communication, and with my definitions of the 3 options you meant to say UDT says 3.
Yes. Sorry for parsing your post incorrectly.
The question is about should we take Bobs utility as a term in our own utility function (and then multiply it by our probabilities), or should we take Bob's expected utility of our action as a term in our own utility function (which already has his probability built into it).
I guess the second option sounds better to me because it generalizes more easily. What if Alice and Bob have diff...
Let us assume for the purpose of this argument that Bayesian probabilities are subjective. Specifically, I am thinking in terms of the model of probability expressed in model 4 here. That is to say that the meaning of P(A)=2/3 is "I as a decision agent care twice as much about possible world in which A is true relative to the possible world in which A is false." Let us also assume that it is possible to have preferences about things that we will never observe.
Consider the following thought experiment:
Alice and Bob are agents who disagree about the fairness of a coin. Alice believes that the coin will come up heads with probability 2/3 and Bob believes the coin will come up tails with probability 2/3. They discuss their reasons for a long time and realize that their disagreement comes from different initial prior assumptions, and they agree that both people have rational probabilities given their respective priors. Alice is given the opportunity to gamble on behalf of Bob. Alice must call heads or tails, then the coin will be flipped once. If Alice calls the coin correctly, then Bob will be given a dollar. If she calls the coin incorrectly, then nothing happens. Either way, nobody sees the result of the coin flip, and Alice and Bob never interact again. Should Alice call heads or tails?
The meat of this question is this: When trying to being altruistic towards a person, should you maximize their expected utility under their priors or your own. I will present an argument here, but feel free to stop reading here and think about it on your own and post the results.
First of all, notice there there are actually 3 options:
1) Maximize your own expectation of Bob's utility
2) Maximize Bob's expectation of his utility
3) Maximize what Bob's expectation of his utility would be if he were to update on the evidence of everything that you have observed.
At first it may have looked like the main options were 1 and 2, but I claim that 2 is actually a very bad option and the only question is between options 1 and 3. Option 2 is stupid because for example it would cause Alice to call tails even if she has already seen the coin flip and it came up heads. There is no reason for Alice not to update on all of the information she has. The only question is whose prior should she update from. In this specific thought experiment, we are assuming that 2 and 3 are the same, since Alice has already convinced herself that her observations could not change Bob's mind, but I think that as a general options 1 and 3 are somewhat reasonable answers, while 2 is not.
Option 3 has the nice property that it does not have to observe Bob's utility function. It only has to observe Bob's expected utility of different choices. This is nice because in many ways "expected utility" seems like a more fundamental and possibly more well defined concept than "utility." We are trying to be altruistic towards Bob. It seems natural to give Bob the most utility in the possible worlds that he "cares about" the most.
On the other hand, we want the possible worlds that we care about most to be as good as possible. We may not ever be able to observe whether of not Bob gets the dollar, but it is not just Bob who wants to get the dollar. We also want Bob to get the dollar. We want Bob to get the dollar in the most important possible worlds, the worlds we assign a high probability to. What we want is for Bob to be happy in the worlds that are important. We may have subjectively assigned those possible worlds to be the most important ones, but from the standpoint of us as a decision agent, the worlds we assign high probability to really are more important than the other ones.
Option 1 is also simpler than option 3. We just have a variable for Bob's utility in our utility function, and we do what maximizes our expected utility. If we took option 3, we would be maximize something that is not just a product of our utilities with our probabilities.
Option 3 has some unfortunate consequences. For example, it might cause us to pray for a religious person even if we are very strongly atheist.
I prefer option 1. I care about the worlds that are simple and therefore are given high probability. I want everyone to be happy in those worlds. I would not sacrifice the happiness in someone in a simple/probable/important world just because someone else thinks another world is important. Probability may be subjective, but relative to the probabilities that I use to make all my decisions, Bob's probabilities are just wrong.
Option 3 is nice in situations where Alice and Bob will continue interacting, possibly even interacting through mutual simulation. If Alice and Bob were given a symmetric scenario, then this would become a prisoner dilemma, where Alice choosing heads corresponds to defecting, while Alice choosing tails corresponds to cooperating. However, I believe this is a separate issue.