Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

The Sleeping Beauty problem and transformation invariances

1 aspera 23 August 2015 08:57PM

I recently read this blog post by Allen Downey in response to a reddit post in response to Julia Galef's video about the Sleeping Beauty problem. Downey's resolution boils down to a conjecture that optimal bets on lotteries should be based on one's expected state of prior information just before the bet's resolution, as opposed to one's state of prior information at the time the bet is made.

I suspect that these two distributions are always identical. In fact, I think I remember reading in one of Jaynes' papers about a requirement that any prior be invariant under the acquisition of new information. That is to say, the prior should be the weighted average of possible posteriors, where the weights are the likelihood that each posterior would be acheived after some measurement. But now I can't find this reference anywhere, and I'm starting to doubt that I understood it correctly when I read it.

So I have two questions:

1) Is there such a thing as this invariance requirement? Does anyone have a reference? It seems intuitive that the prior should be equivalent to the weighted average of posteriors, since it must contain all of our prior knowledge about a system. What is this property actually called?

2) If it exists, is it a corollary that our prior distribution must remain unchanged unless we acquire new information?

Comment author: Kurros 26 November 2013 12:27:48AM 2 points [-]

You don't think people here have a term for their survey-completing comrades in their cost function? Since I probably won't win either way this term dominated my own cost function, so I cooperated. An isolated defection can help only me, whereas an isolated cooperation helps everyone else and so gets a large numerical boost for that reason.

Comment author: aspera 26 November 2013 04:08:28AM 1 point [-]

It's true: if you're optimizing for altruism, cooperation is clearly better.

I guess it's not really a "dilemma" as such, since the optimal solution doesn't depend at all on what anyone else does. If you're trying to maximize EV, defect. If you're trying to maximize other people's EV, cooperate.

Comment author: EGI 24 November 2013 06:20:12PM 20 points [-]

Surveyed, including bonus.

I really liked the monetary reward prisoners dillema. I am really curious how this turns out. Given the demographic here, I would predict ~ 85% cooperate.

The free text options were rendered in german (Sonstige). Was that a bug or does it serve some hidden purpose?

Comment author: aspera 25 November 2013 06:05:49PM 4 points [-]

My confidence bounds were 75% and 98% for defect, so my estimate was diametrically opposed to yours. If the admittedly low sample size of these comments is any indication, we were both way off.

Why do you think most would cooperate? I would expect this demographic to do a consequentialist calculation, and find that an isolated cooperation has almost no effect on expected value, whereas an isolated defection almost quadruples expected value.

Comment author: aspera 25 November 2013 06:02:23PM 12 points [-]

Nice job on the survey. I loved the cooperate/defect problem, with calibration questions.

I defected, since a quick expected value calculation makes it the overwhelmingly obvious choice (assuming no communcation between players, which I am explicitly violating right now). Judging from comments, it looks like my calibration lower bound is going to be way off.

Comment author: wedrifid 16 October 2013 03:27:34AM 0 points [-]

Unfortunately, I think that ascribes too much power to VNM utility functions (that term itself is a LessWrongism; elsewhere, they would be called cardinal utility functions or just utility functions).

I actually don't recall seeing the usage "VNM utility functions" on less wrong at all, prior to this thread. It may have occurred previously but certainly not with sufficient frequency as to be a 'lesswrongism'. As you say, the "VNM" is unnecessary in that context since is all the VNM part does is say "it must have a utility function because it adheres to these axioms".

It is sometimes necessary to explicitly refer to things other than 'utility functions' with a 'VNM' qualifier. This is largely to pre-empt pedants who, when reading unqualified usage 'consequentialist', are not willing to assume that it refers to the only kind of consequentialist that is ever significantly discussed here (those that have utility functions).

VNM utility also falls flat on its face unless we already know what we prefer.

Not quite, but the point stands. The actual requirement is that there is any way to collect any evidence at all about our preferences (or, to be even more general, any way to cause outcomes to be correlated to our preference).

Comment author: aspera 16 October 2013 03:06:42PM 0 points [-]

For the moment, I'm going to strike the comment from the post. I don't want to ascribe a viewpoint to VincentYu that he doesn't actually hold.

Comment author: VincentYu 13 October 2013 04:54:35PM *  7 points [-]

The answer [edited Oct 13, 2013]

As several commenters have pointed out, the original problem does not supply a method for taking limits. Our analysis shows that the problem is ill posed: it has no unique solution unless we take on additional assumptions.

I disagree that the mathematics of original problem is ill-posed, and I think DanielLC made the same point. The point of contention seems to center on the use of infinities in the original problem, which is indeed an issue if they were manipulated as real numbers, but they were not. It is perfectly acceptable and mathematical to have a countably infinite set of objects, and to define a sequence of subsets corresponding to the time evolution of that set. Infinite sets are not defined as the limit of some sequence of finite sets! There is no ambiguity in the mathematics of original problem.*

Because the use of infinity in the original problem is not in the sense of a limit, there is no good reason to think that we should take limits, or that the limits of the solutions to the finite problems should correspond in any way to the solutions of the original problem.

Where there are ambiguities are in the use of the word "utility" and similar concepts as though they were well-defined in this context. And in this sense, I agree that the original problem is ill-posed.

* There are mathematical ambiguities in an unfavorable reading of the original problem, but the following steelman removes them: Biject the people with the natural numbers, and then transfer the nth person on day n.

Comment author: aspera 15 October 2013 07:03:33PM 1 point [-]

I added a section called "Deciding how to decide" that (hopefully) deals with this issue appropriately. I also amended the conclusion, and added you as an acknowledgement.

Comment author: D_Alex 15 October 2013 03:36:09AM 0 points [-]

I'd prefer to see this in Main, it is interesting and important.

Comment author: aspera 15 October 2013 03:45:35AM 0 points [-]

I'm not sure why it got moved: maybe not central to the thesis of LW, or maybe not high enough quality. I'm going to add some discussion of counter-arguments to the limit method. Maybe that will make a difference.

I noticed that the discussion picked up when it got moved, and I learned some useful stuff from it, so I'm not complaining.

Comment author: VincentYu 13 October 2013 09:46:20PM *  1 point [-]

Tell me if this is right:

It is possible mathematically to represent a countably infinite number of immortal people, as well as the process of moving them between spheres. Further, we should not expect a priori that a problem involving such infinities would have a solution equivalent to those solutions reached by taking infinite limits of an analogous finite problem.

That's an accurate interpretation of my comment.

Some confusion arises when we introduce the concept of “utility” to determine which of the two choices is better, since utility only serves as a basis on which to make decision for finite problems.

I do think that confusion arises in this context from the concept of "utility", but not because "utility only serves as a basis on which to make decision for finite problems." The "utility" in the problem is clearly not that of VNM-utility (of which I previously gave a brief explanation) because we not assigning utility to actions, decisions, or choices (a VNM-utility function U would generally have no problem responding to an infinite set of choices, as it simply says: do argmax_{choice}(U(choice))). This severely undermines what we can do with the "utility" in the problem because we are left with the various flavors of aggregative utilitarianism, which suffer from intractable problems even in finite situations! Attempting to extend them to the situation at hand is problematic (and, as Kaj_Sotala remarked, dealing with infinities in aggregative consequentialism is the topic of one of Bostrom's papers).

  1. Do you view the paradox as therefore unresolvable as stated, or would you claim that a different resolution is correct?

I think that the appearance of the paradox is a consequence of unfamiliarity with infinite sets, and that it is not too surprising that our intuition appears to contradict itself in this context (by presenting each option as better than the other). The contradictory intuitions don't correspond to a logical contradiction, so the apparent paradox needs no resolution. The actual problem (choosing between the two options) is a matter of preference, just as the choice between strawberry and chocolate is a matter of preference.

2. If I carefully restricted my claim about ill-posedness to the question of which choice is better from a utilitarian sense, would you agree with it?

Absolutely. I think aggregative utilitarianism (as a moral theory) is screwed even in finite scenarios, much less infinite scenarios. (But I also think aggregative utilitarianism is a good but ill-defined general standard for comparing consequences in real life.)

Comment author: aspera 14 October 2013 03:11:52AM 1 point [-]

Ok, I think I've got it. I'm not familiar with VNM utility, and I'll make sure to educate myself.

I'm going to edit the post to reflect this issue, but it may take me some time. It is clear (now that you point it out) that we can think of the ill-posedness coming from our insistence that the solution conform to aggregative utilitarianism, and it may be possible to sidestep the paradox if we choose another paradigm of decision theory. Still, I think it's worth working as an example, because, as you say, AU is a good general standard, and many readers will be familiar with it. At the minimum, this would be an interesting finite AU decision problem.

Thanks for all the time you've put into this.

Comment author: VincentYu 13 October 2013 04:54:35PM *  7 points [-]

The answer [edited Oct 13, 2013]

As several commenters have pointed out, the original problem does not supply a method for taking limits. Our analysis shows that the problem is ill posed: it has no unique solution unless we take on additional assumptions.

I disagree that the mathematics of original problem is ill-posed, and I think DanielLC made the same point. The point of contention seems to center on the use of infinities in the original problem, which is indeed an issue if they were manipulated as real numbers, but they were not. It is perfectly acceptable and mathematical to have a countably infinite set of objects, and to define a sequence of subsets corresponding to the time evolution of that set. Infinite sets are not defined as the limit of some sequence of finite sets! There is no ambiguity in the mathematics of original problem.*

Because the use of infinity in the original problem is not in the sense of a limit, there is no good reason to think that we should take limits, or that the limits of the solutions to the finite problems should correspond in any way to the solutions of the original problem.

Where there are ambiguities are in the use of the word "utility" and similar concepts as though they were well-defined in this context. And in this sense, I agree that the original problem is ill-posed.

* There are mathematical ambiguities in an unfavorable reading of the original problem, but the following steelman removes them: Biject the people with the natural numbers, and then transfer the nth person on day n.

Comment author: aspera 13 October 2013 08:23:40PM 1 point [-]

I would like to include this issue in the post, but I want to make sure I understand it first. Tell me if this is right:

It is possible mathematically to represent a countably infinite number of immortal people, as well as the process of moving them between spheres. Further, we should not expect a priori that a problem involving such infinities would have a solution equivalent to those solutions reached by taking infinite limits of an analogous finite problem. Some confusion arises when we introduce the concept of “utility” to determine which of the two choices is better, since utility only serves as a basis on which to make decision for finite problems.

If that’s what you’re saying, I have a couple of questions.

  1. Do you view the paradox as therefore unresolvable as stated, or would you claim that a different resolution is correct?

  2. If I carefully restricted my claim about ill-posedness to the question of which choice is better from a utilitarian sense, would you agree with it?

Comment author: aspera 13 October 2013 04:16:09PM 0 points [-]

The final section has been edited to reflect the concerns of some of the commenters.

View more: Next