Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

VincentYu comments on Counterfactual self-defense - Less Wrong Discussion

0 Post author: MrMind 23 November 2012 10:15AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (8)

You are viewing a single comment's thread. Show more comments above.

Comment author: MrMind 23 November 2012 03:58:02PM 0 points [-]

Dialogue 2 and dialogue 3 as you phrased them are equivalent, but they both omit a significant aspect of the original discussion -- that Omega promises that if the coin had come up tails, it would have offered the same bet (which would now have been a winning one for you).

Mmm... I implied it by saying in the first paragraph that Omega is reliable and that's common knowledge, but it's true that the wording could have been much clearer. I wonder if an edit would do more harm than good.

Your scenarios, as stated, leave it unclear whether the bet is offered because it came up heads. So the possibility is left open that Omega only offer bets when he knows that the coin came up heads.

That's on purpose, and is exactly the information that the agent doesn't have in one-shot counterfactual mugging.

Taking the scenario where we know that Omega would have offered the bet regardless of what the coin-toss was: that effectively means that, in this type of decisions, statistically speaking, agents are favoured who exhibit some sort of "timelessness" in their decision theory, agents are favoured who do not update in the sense that a CDT agent would update.

So to have that "winning decision theory" (which is winning overall, not in individual cases), we must be agents who do not update in this manner.

That is all well estabilished, I think. Yet:

The problem people tend to have with this is that they seem to assume a winning decision theory to be one which maximizes the expected utility of each of any individual decisions as if they're logically independent from each other, but in reality we want a decision theory that maximizes the expected summed utility over the whole life-length of the decision theory.

A DT though that would accept only winning counterfactual but behaves in every other occasion like a TDT agent is even better. My problem is understanding if the rule proposed is consistent with this desiderata, or would turn a TDT back into a CDT.

Comment author: VincentYu 23 November 2012 04:32:14PM *  4 points [-]

Dialogue 2 and dialogue 3 as you phrased them are equivalent, but they both omit a significant aspect of the original discussion -- that Omega promises that if the coin had come up tails, it would have offered the same bet (which would now have been a winning one for you).

Mmm... I implied it by saying in the first paragraph that Omega is reliable and that's common knowledge, but it's true that the wording could have been much clearer. I wonder if an edit would do more harm than good.

Common knowledge of Omega's reliability is not sufficient. ArisKatsaris is pointing out that in your current post, Omega is allowed to condition its offer on the outcome of the coin flip. The original discussion on counterfactual mugging specifies that this is not allowed—Omega's offer is independent of the coin flip.

Your scenarios, as stated, leave it unclear whether the bet is offered because it came up heads. So the possibility is left open that Omega only offer bets when he knows that the coin came up heads.

That's on purpose, and is exactly the information that the agent doesn't have in one-shot counterfactual mugging.

You do have the information. In counterfactual mugging, Omega tells you truthfully (with its trustworthiness being common knowledge) that it would have given the same offer if the coin had landed differently.

Comment author: MrMind 26 November 2012 02:42:40PM 0 points [-]

Common knowledge of Omega's reliability is not sufficient. ArisKatsaris is pointing out that in your current post, Omega is allowed to condition its offer on the outcome of the coin flip. The original discussion on counterfactual mugging specifies that this is not allowed—Omega's offer is independent of the coin flip.

You do have the information. In counterfactual mugging, Omega tells you truthfully (with its trustworthiness being common knowledge) that it would have given the same offer if the coin had landed differently.

I'm starting to think that there's a deeper-than-I-thought point about the extraction of this information from the way I structured the dialogue. If all that Omega offers is a series of bet, and the TDT agent has no information about what would have Omega done if the coin toss came out differently or even if he will see Omega again, then it's not clear to me what a TDT agent should do.

Comment author: ArisKatsaris 26 November 2012 04:12:53PM 1 point [-]

If all that Omega offers is a series of bet, and the TDT agent has no information about what would have Omega done if the coin toss came out differently or even if he will see Omega again, then it's not clear to me what a TDT agent should do.

Indeed. I think that's why it's sometimes better to imagine "Omega" as some sort of stable physical process whose complete functionality we know, instead of an as an agent with mysterious motivations.