Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Tyrrell_McAllister 22 July 2014 08:39:07PM 1 point [-]

[Resuming my tongue-in-cheek argument...]

It is true that adding different utility functions is in general an error. However, for agents bound to follow Rationality (and Rationality alone), the different utility functions are best thought of as the same utility function conditioned on different hypotheses, where the different hypotheses look like "The utility to P2 turns out to be what really matters".

After all, if the agents are making their decisions on the basis of Rationality alone, then Rationality alone must have a utility function. Since Rationality is universal, the utility function must be universal. What alternative does Rationality have, given the constraints of the problem, other than a weighted sum of the utility functions of the different individuals who might turn out to matter?

Comment author: JGWeissman 22 July 2014 08:50:43PM 0 points [-]

"Rationality" seems to give different answer to the same problem posed with different affine transformations of the players' utility functions.

Comment author: Tyrrell_McAllister 22 July 2014 06:00:20PM *  2 points [-]

[The following argument is made with tongue somewhat in cheek.]

Rationality (with a capital R) is supposed to be an idealized algorithm that is universally valid for all agents. Therefore, this algorithm, as such, doesn't know whether it will be instantiated in Player One (P1) or Player Two (P2). Yes, each player knows which one they are. But this knowledge is input that they feed to their Rationality subroutine. The subroutine itself doesn't come with this knowledge built in.

Since Rationality doesn't know where it will end up, it doesn't know which outcomes will maximize its utility. Thus, the most rational thing for Rationality (qua idealized algorithm) to do is to precommit to strategies for P1 and P2 that maximize its expected utility given this uncertainty.

That is, let EU₁(s, t) be the expected utility to P1 if P1 implements strategy s and P2 implements strategy t. Define EU₂(s, t) similarly. Then Rationality wants to precommit the players to the respective strategies s and t that maximize

p EU₁(s, t) + q EU₂(s, t),

where p (respectively, q) is the measure of the copies of Rationality that end up in P1 (respectively, P2).

Assuming that p = q, Rationality will therefore have P1 choose C (with probability 1) and have P2 choose Y (with probability 1).

That's all well and good, but will the players actually act that way? After all, they are stipulated to care only about themselves, so P2 in particular will not want to act according to this selfless strategy.

Yes, but this selfish desire on P2's part is not built into P2's Rationality subroutine (call it R), because Rationality is universal. P2's selfish desires must be implemented elsewhere, outside of R. To be sure, P2 is free to feed the information that it is P2 to R, but R won't do anything with this information, because R is already precommitted to a strategy for the reasons given above.

And since the players are given to be rational, they are forced to act according to the strategies pre-selected by their Rationality subroutines, despite their wishes to the contrary. Therefore, they will in fact act as Rationality determined.

[If this comment has any point, it is that there is a strong tension, if not a contradiction, between the idea of rationality as a universally valid mode of reasoning, on the one hand, and the idea of rational agents whose revealed preferences are selfish, on the other.]

Comment author: JGWeissman 22 July 2014 07:21:32PM *  1 point [-]

Error: Adding values from different utility functions.

See this comment.

Comment author: itaibn0 21 July 2014 06:35:31PM 1 point [-]

You're right. I'm not actually advocating this option. Rather, I was comparing EY's seemingly arbitrary strategy with other seemingly arbitrary strategies. The only one I actually endorse is "P1: A". It's true that this specific criterion is not invariant under affine transformations of utility functions, but how do I know EY's proposed strategy wouldn't change if we multiply player 2's utility function by 100 as you propose?

(Along a similar vein, I don't see how I can justify my proposal of "P1: 3/10 C 7/10 B". Where did the 10 come from? "P1: 2/7 C 5/7 B" works equally well. I only chose it because it is convenient to write down in decimal.)

Comment author: JGWeissman 21 July 2014 07:10:24PM 3 points [-]

Eliezer's "arbitrary" strategy has the nice property that it gives both players more expected utility than the Nash equilibrium. Of course there are other strategies with this property, and indeed multiple strategies that are not themselves dominated in this way. It isn't clear how ideally rational players would select one of these strategies or which one they would choose, but they should choose one of them.

Comment author: itaibn0 21 July 2014 04:53:01PM 2 points [-]

I have no idea where those numbers came from. Why not "P1: .3C .7B" to make "P2: Y" rational? Otherwise, why does P2 play Y at all? Why not "P1: C, P2: Y", which maximizes the sum of the two utilities, and is the optimal precommitment under the Rawlian veil-of-ignorance prior? Heck, why not just play the unique Nash equilibrium "P1: A"? Most importantly, if there's no principled way to make these decisions, why assume your opponent will timelessly make them the same way?

Comment author: JGWeissman 21 July 2014 05:28:00PM 2 points [-]

Why not "P1: C, P2: Y", which maximizes the sum of the two utilities, and is the optimal precommitment under the Rawlian veil-of-ignorance prior?

If we multiply player 2's utility function by 100, that shouldn't change anything because it is an affine transformation to a utility function. But then "P1: B, P2: Y" would maximize the sum. Adding values from different utility functions is a meaningless operation.

Comment author: James_Miller 20 July 2014 11:31:07PM 0 points [-]

"I also disagree that player 1 not picking A provides useful information to player 2."

Player 1 gets 3 if he picks A and 2 if he picks B, so doesn't knowing that Player 1 did not pick A provide useful information as to whether he picked B?

Comment author: JGWeissman 21 July 2014 01:00:42AM 0 points [-]

The reason player 1 would choose B is not because it directly has a higher payout but because including B in a mixed strategy gives player 2 an incentive to include Y in its own mixed strategy, increasing the expected payoff of C for player 1. The fact that A dominates B is irrelevant. The fact that A has better expected utility than the subgame with B and C indicates that player 1 not choosing A is somehow irrational, but that doesn't give a useful way for player 2 to exploit this irrationality. (And in order for this to make sense for player 1, player 1 would need a way to counter exploit player 2's exploit, and for player 2 to try its exploit despite this possibility.)

Comment author: James_Miller 20 July 2014 10:50:07PM *  2 points [-]

Let's try to find the source of our disagreement. Would you agree with the following:

"You can only have a subgame that excludes A if the fact that Player 1 has not picked A provides no useful information to Player 2 if Player 2 gets to move."

Comment author: JGWeissman 20 July 2014 11:10:17PM 0 points [-]

The definition you linked to doesn't say anything about entering subgame not giving the players information, so no, I would not agree with that.

I would agree that if it gave player 2 useful information, that should influence the analysis of the subgame.

(I also don't care very much whether we call this object within the game of how the strategies play out given that player 1 doesn't choose A a "subgame". I did not intend that technical definition when I used the term, but it did seem to match when I checked carefully when you objected, thinking that maybe there was a good motivation for the definition so it could indicated a problem with my argument if it didn't fit.)

I also disagree that player 1 not picking A provides useful information to player 2.

Comment author: James_Miller 20 July 2014 09:34:32PM *  -1 points [-]

I'm sorry but "subgame" has a very specific definition in game theory which you are not being consistent with. Also, intuitively when you are in a subgame you can ignore everything outside of the subgame, playing as if it didn't exist. But when Player 2 moves he can't ignore A because the fact that Player 1 could have picked A but did not provides insight into whether Player 1 picked B or C. I am a game theorist.

Comment author: JGWeissman 20 July 2014 10:43:56PM 2 points [-]

I'm sorry but "subgame" has a very specific definition in game theory which you are not being consistent with.

I just explained in detail how the subgame I described meets the definition you linked to. If you are going to disagree, you should be pointing to some aspect of the definition I am not meeting.

Also, intuitively when you are in a subgame you can ignore everything outside of the subgame, playing as if it didn't exist. But when Player 2 moves he can't ignore A because the fact that Player 1 could have picked A but did not provides insight into whether Player 1 picked B or C.

If it is somehow the case that giving player 2 info about player 1 is advantageous for player 1, then player 2 should just ignore the info, and everything still plays out as in my analysis. If it is advantageous for player 2, then it just strengthens the case that player 1 should choose A.

I am a game theorist.

I still think you are making a mistake, and should pay more attention to the object level discussion.

Comment author: James_Miller 20 July 2014 08:26:36PM *  -1 points [-]

"Classical game theory says that player 1 should chose A for expected utility 3, as this is better than than the sub game of choosing between B and C "

No since this is not a subgame because of the uncertainty. From Wikipedia " In game theory, a subgame is any part (a subset) of a game that meets the following criteria...It has a single initial node that is the only member of that node's information set... "

I'm uncertain about what TDT/UDT would say.

Comment author: JGWeissman 20 July 2014 09:24:59PM 1 point [-]

To see that it is indeed a subgame:

Represent the whole game with a tree whose root node represents player 1 choosing whether to play A (leads to leaf node), or to enter the subgame at node S. Node S is the root of the subgame, representing player 1's choices to play B or C leading to nodes representing player 2 choice to play X or Y in those respective cases, each leading to leaf nodes.

Node S is the only node in its information set. The subgame contains all the descendants of S. The subgame contains all nodes in the same information set as any node in the subgame. It meets the criteria.

There is no uncertainty that screws up my argument. The whole point of talking about the subgame was to stop thinking about the possibility that player 1 chose A, because that had been observed not to happen. (Of course, I also argue that player 2 should be interested in logically causing player 1 not to have chosen A, but that gets beyond classical game theory.)

Comment author: JGWeissman 20 July 2014 08:20:39PM 9 points [-]

Classical game theory says that player 1 should chose A for expected utility 3, as this is better than than the sub game of choosing between B and C where the best player 1 can do against a classically rational player 2 is to play B with probability 1/3 and C with probability 2/3 (and player 2 plays X with probability 2/3 and Y and with probability 1/3), for an expected value of 2.

But, there are pareto improvements available. Player 1's classically optimal strategy gives player 1 expected utility 3 and player 2 expected utility 0. But suppose instead Player 1 plays C, and player 2 plays X with probability 1/3 and Y with probability 2/3. Then the expected utility for player 1 is 4 and for player 2 it is 1/3. Of course, a classically rational player 2 would want to play X with greater probability, to increase its own expected utility at the expense of player 1. It would want to increase the probability beyond 1/2 which is the break even point for player 1, but then player 1 would rather just play A.

So, what would 2 TDT/UDT players do in this game? Would they manage to find a point on the pareto frontier, and if so, which point?

Comment author: shminux 02 July 2014 04:23:08PM *  3 points [-]

No need for a conflict or a ban, just let them know that their user name will be made public.

I find it quite uncomfortable to initiate conflict with people.

Not sure why the parent is upvoted. If you have trouble confronting people, you make a poor admin. Is there another active admin on LW who is more competent?

EDIT: I assumed too much, Kaj was probably not expected to moderate and ended up in this position by default. Sorry.

Comment author: JGWeissman 02 July 2014 04:41:55PM 29 points [-]

If you have trouble confronting people, you make a poor admin.

Can we please act like we actually know stuff about practical instrumental rationality given how human brains work, and not punish people for openly noticing their weaknesses.

You could have more constructively said something like "Thank you for taking on these responsibilities even though it sometimes makes you uncomfortable. I wonder if anyone else who is more comfortable with that would be willing to help out."

View more: Next