Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Conventions and Confusing Continuity Conundrums

2 Post author: Psy-Kosh 01 May 2009 01:41AM

The next "How Not to be Stupid" may be a bit delayed for a couple of reasons.

First, there appears to be a certain unstated continuity assumption in the material I've been working from that would probably be relevant for the next posting. As I said in the intro post, I'm working from Stephen Omhunduro's vulnerability argument, but filling in what I viewed as missing bits, generalizing one or two things, and so on. Anyways, the short of it is I thought that I was able to how to derive the relevant continuity condition, but turns out I need to think about that a bit more carefully.

If I remain stumped on that bit, I'll just post and explicitly state the assumption, pointing it out as a potential problem area that needs to be dealt with one way or another. ie, either solved somehow, or demonstrate that it actually is invalid (thus causing some issues for decision theory...)

Also, I'm going to be at Penguicon the next few days.

Actually, I think I'll right now state the continuity condition I need, let others play with it too:

Basically, I need it to be that if there's a preference ranking A > C > B, there must exist some p such that the p*A + (1-p)*B lottery ranks equal to C. (That is, that the mixing lotteries correspond to a smooth spectrum of preferences between, well, the things they are a mixing of rather than having any discontinuous jumps.)

Anyways, I hope we can put this little trouble bit behind us and resume climbing the shining path of awakening to nonstupidity. :)

Comments (9)

Comment author: Eliezer_Yudkowsky 01 May 2009 06:29:17AM 3 points [-]

If a dust speck in the eye is worse than nothing, and being tortured for 50 years is worse than a dust speck, there must be some probability of being tortured for 50 years which is so small that you are indifferent between that and a certainty of getting a dust speck in the eye? I quite agree!

See you at Penguicon...

Comment author: MendelSchmiedekamp 01 May 2009 02:29:22PM 2 points [-]

Your link deals neither with equality of preferences or with probability. Could you please explain its relevance?

Also, why does this example imply that continuity is generally valid?

Comment author: Eliezer_Yudkowsky 01 May 2009 05:32:11PM 0 points [-]

It's about continuity and quantitative commensurability of preferences. Aggregating lots of small events is not quite the same balancing method as multiplying a large event by a tiny probability, and I think some people did bite the second bullet but not the first (?!) - but it's the same basic concept of continuity and quantitative commensurability that lets you compare utility intervals on a common scale and "shut up and multiply".

Comment author: MendelSchmiedekamp 01 May 2009 06:04:57PM *  1 point [-]

The commonality between aggregation and probability for multiplying that is occurring in this case is reasonable, but that lies on the same level as the argument that Psy-Kosh makes in adorable maybes.

The point of this post is that his argument doesn't just give you continuity. There is some missing step.

Elsewhere in these comments I'm claiming what is missing is actually an additional premise.

Comment author: Psychohistorian 01 May 2009 12:36:53PM 2 points [-]

The actual effort it requires to consider which option to take is itself probably less pleasant than just taking the dust speck. Of course, this decision cost doesn't apply to models, but it's worth considering if you're thinking of actually being faced with the decision yourself.

Comment author: MendelSchmiedekamp 01 May 2009 02:35:22AM 1 point [-]

Assume that your agents are either omniscient beings with unbounded computation or oracle enabled (1), and assume that the lottery is instantenous (2).

For continuity of preference rankings, the following are required:

a) total ordering of preferences, and b) continuity of the mapping of p into the preference space, and c) p ranges over a closed set with closure under sequences.

Brief Sketch: Total ordering is needed because this ensures that the pA + (1-p)B for various p is totally ordered. Combined with the continuity of the p map into preference space this ensures that pA + (1-p)B can get arbitrarilly close to C (and A and B). By the closure property (and closedness of the set) the limits of all sequences are contained in the set, so since C is the limit for some sequence of p'A + (1-p')B, such a p' must exist in the set.

Each of these requirements is necessary. Lacking total ordering can mean that all pA + (1-p)B are incomparable, so no sequence can be generated to get to C. Lacking continuity can mean that no pA + (1-p)B are within some arbitrary bound of C, hence no limit point would exist. Lacking the closure property means that you may get arbitrarilly close, but cannot actually equal C.

1 - This ensures that an ordering, total or otherwise can be constructed which is either non-self-referential (because all preferences are final for an omniscient agent) or safely self-referential (using oracles of the appropriate types).

2 - This ensures that the lottery does not create a distinct preference state, and serves only to trade-off between the two original states.

Comment author: Psy-Kosh 01 May 2009 03:19:08AM *  0 points [-]

Thanks. Though the continuity assumption is itself the thing I felt to be the problem area. Unless I'm misunderstanding your argument, you're assuming the very continuity property I want to derive.

(Incidentally, I may be wrong on this, but I think the closure property you're referring to would follow directly from the continuity. (alternately, one might need to show that closure property to show the continuity. Point being, I don't think those two properties are separable, except at A and B))

Comment author: Douglas_Knight 01 May 2009 04:37:06AM *  1 point [-]

Continuity is merely preserving order: if A > B and p > q, then pA+(1-p)B > qA+(1-q)B. It is a not-being-stupid assumption. or an interpretation of probability.

Mendel seems to be working in an extremely abstract version of probability where p cannot be described as a size. But once you insist on p being a number, there are many possibilities. You might allow p only to take rational values, so that they can be finitely represented. Or you might allow p to take all real values, in which case some p exists solving the problem.

The issue of closure is about where the p's live, that is, what kinds of lotteries you can build. It isn't about preferences or states of the world (except in that lotteries are states of the world).

ETA: Actually...the axiom of independence gives you order preservation. The axiom of continuity does only one thing: it rules out lexicographic preferences. It says that if you lexicographically care more about X than about Y, you aren't allowed to use Y as a tie breaker, but must simply not care about about Y at all.

Comment author: MendelSchmiedekamp 01 May 2009 04:06:43AM *  1 point [-]

The continuity of p -> preference space is an additional property you must assert. Like the total ordering, you need to specify that this function is in fact of this form, since it isn't determined by the premises

A monotonically increasing function from p -> preference space will preserve the total order just fine and meet the end point criteria, even if discontinuous. And since those are the two properties you currently require you need to add another property to the theory if you want to eliminate discontinuity.

Intuitively, you probably want to do this anyway, since you haven't said anything about how decisions are made except by listing basic decision pathologies and ruling them out. First was cycles (although a total order is arguably over-kill for this one). The second was incorporating probability (which necessitated an end-point condition - A >= pA + (1-p)B >= B). Now it's time to add a new condition of continuity of the p mapping, based on the intuition that immeasurably small changes in probability should not cause measurable changes in decisions. But nothing you've laid out so far excludes such an agent.

(Good point. My error there. But do note, though, while b implies c, c definitely does not imply b.)

Edited to add: The intuition about measureable changes caused by immeasurable probability shifts removes all but point-wise discontinuitites. Those you can remove via adding something like c, i.e. probabilities are real numbers or the like.