= 27d567bc2d48d9f40e9ccc20ea370b15)
snarles comments on Probabilities Small Enough To Ignore: An attack on Pascal's Mugging - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (176)
I'll need some background here. Why aren't bounded utilities the default assumption? You'd need some extraordinary arguments to convince me that anyone has an unbounded utility function. Yet this post and many others on LW seem to implicitly assume unbounded utility functions.
1) We don't need an unbounded utility function to demonstrate Pascal's Mugging. Plain old large numbers like 10^100 are enough.
2) It seems reasonable for utility to be linear in things we care about, e.g. human lives. This could run into a problem with non-uniqueness, i.e., if I run an identical computer program of you twice, maybe that shouldn't count as two. But I think this is sufficiently murky as to not make bounded utility clearly correct.
The scale is arbitrary. If your utility function is designed such that utility for common scenario are not very small compared to the maximum utility then you wouldn't have Pascal's Muggings.
Does anybody really have linear preferences in anything? This seems at odds with empirical evidence.
Like V_V, I don't find it "reasonable" for utility to be linear in things we care about.
I will write a discussion topic about the issue shortly.
EDIT: Link to the topic: http://lesswrong.com/r/discussion/lw/mv3/unbounded_linear_utility_functions/
Because here the default utility is the one specified by the Von Neumann-Morgenstern theorem and there is no requirement (or indication) that it is bounded.
Humans, of course, don't operate according to VNM axioms, but most of LW thinks it's a bug to be fixed X-/
Except, the VNM theorem in the form given applies to situations with finitely many possibilities. If there are infinitely many possibilities, then the generalized theorem does require bounded utility. This follows from precisely the Pascal's mugging-type arguments like the ones being considered here.
(And with finitely many possibilities, the utility function cannot possibly be unbounded, because any finite set of reals has a maximum.)
In the page cited a proof outline is given for the finite case, but the theorem itself has no such restriction, whether "in the form given" or, well, the theorem itself.
What are you referring to as the generalised theorem? Something other than the one that VNM proved? That certainly does not require or assume bounded utility.
If you're referring to the issue in the paper that entirelyuseless cited, Lumifer correctly pointed out that it is outside the setting of VNM (and someone downvoted him for it).
The paper does raise a real issue, though, for the setting it discusses. Bounding the utility is one of several possibilities that it briefly mentions to salvage the concept.
The paper is also useful in clarifying the real problem of Pascal's Mugger. It is not that you will give all your money away to strangers promising 3^^^3 utility. It is that the calculation of utility in that setting is dominated by extremes of remote possibility of vast positive and negative utility, and nowhere converges.
Physicists ran into something of the sort in quantum mechanics, but I don't know if the similarity is any more than superficial, or if the methods they worked out to deal with it have any analogue here.
Try this:
Theorem: Using the notation from here, except we will allow lotteries to have infinitely many outcomes as long as the probabilities sum to 1.
If an ordering satisfies the four axioms of completeness, transitivity, continuity, and independence, and the following additional axiom:
Axiom (5): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1 and N >= Sum(i=0...n, p_i M_i)/Sum(i=0...n, p_i) then N >= L. And similarly with the arrows reversed.
An agent satisfying axioms (1)-(5) has preferences given by a bounded utility function u such that, L>M iff Eu(L)>Eu(M).
Edit: fixed formatting.
That appears to be an axiom that probabilities go to zero enough faster than utilities that total utility converges (in a setting in which the sure outcomes are a countable set). It lacks something in precision of formulation (e.g. what is being quantified over, and in what order?) but it is fairly clear what it is doing. There's nothing like it in VNM's book or the Wiki article, though. Where does it come from?
Yes, in the same way that VNM's axioms are just what is needed to get affine utilities, an axiom something like this will give you bounded utilities. Does the axiom have any intuitive appeal, separate from it providing that consequence? If not, the axiom does not provide a justification for bounded utilities, just an indirect way of getting them, and you might just as well add an axiom saying straight out that utilities are bounded.
None of which solves the problem that entirelyuseless cited. The above axiom forbids the Solomonoff prior (for which pi Mi grows with busy beaver fastness), but does not suggest any replacement universal prior.
No, the axiom doesn't put any constraints on the probability distribution. It merely constrains preferences, specifically it says that preferences for infinite lotteries should be the 'limits' of the preference for finite lotteries. One can think of it as a slightly stronger version of the following:
Axiom (5'): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1. Then if for all i N>=M_I then N>=L. And similarly with the arrows reversed. (In other words if N is preferred over every element of a lottery then N is preferred over the lottery.)
In fact, I'm pretty sure that axiom (5') is strong enough, but I haven't worked out all the details.
Sorry, there were some formatting problems, hopefully it's better now.
The M_i's are lotteries that the agent has preferences over, not utility values. Thus it doesn't a priori make sense to talk about its growth rate.
I think I understand what the axiom is doing. I'm not sure it's strong enough, though. There is no guarantee that there is any N that is >= M_i for all i (or for all large enough i, a weaker version which I think is what is needed), nor an N that is <= them. But suppose there are such an upper Nu and a lower Nl, thus giving a continuous range between them of Np = p Nl + (1-p) Nu for all p in 0..1. There is no guarantee that the supremum of those p for which Np is a lower bound is equal to the infimum of those for which it is an upper bound. The axiom needs to stipulate that lower and upper bounds Nl and Nu exist, and that there is no gap in the behaviours of the family Np.
One also needs some axioms to the effect that a formal infinite sum Sum{i>=0: pi Mi} actually behaves like one, otherwise "Sum" is just a suggestively named but uninterpreted symbol. Such axioms might be invariance under permutation, equivalence to a finite weighted average when only finitely many pi are nonzero, and distribution of the mixture process to the components for infinite lotteries having the same sequence of component lotteries. I'm not sure that this is yet strong enough.
The task these axioms have to perform is to uniquely extend the preference relation from finite lotteries to infinite lotteries. It may be possible to do that, but having thought for a while and not come up with a suitable set of axioms, I looked for a counterexample.
Consider the situation in which there is exactly one sure-thing lottery M. The infinite lotteries, with the axioms I suggested in the second paragraph, can be identified with the probability distributions over the non-negative integers, and they are equivalent when they are permutations of each other. All of the distributions with finite support (call these the finite lotteries) are equivalent to M, and must be assigned the same utility, call it u. Take any distribution with infinite support, and assign it an arbitrary utility v. This determines the utility of all lotteries that are weighted averages of that one with M. But that won't cover all lotteries yet. Take another one and give it an arbitrary utility w. This determines the utility of some more lotteries. And so on. I don't think any inconsistency is going to arise. This allows for infinitely many different preference orderings, and hence infinitely many different utility functions.
The construction is somewhat analogous to constructing an additive function from reals to reals, i.e. one satisfying f(a+b) = f(a) + f(b). The only continuous additive functions are multiplication by a constant, but there are infinitely many non-continuous additive functions.
An alternative approach would be to first take any preference ordering consistent with the axioms, then use the VNM axioms to construct a utility function for that preference ordering, and then to impose an axiom about the behaviour of that utility function, because once we have utilities it's easy to talk about limits. The most straightforward such axiom would be to stipulate that U( Sum{i>=0: pi Mi} ) = Sum{i>=0: pi U(Mi)}, where the sum on the right hand side is an ordinary infinite sum of real numbers. The axiom would require this to converge.
This axiom has the immediate consequence that utilities are bounded, for if they were not, then for any probability distribution {i>=0: pi} with infinite support, one could choose a sequence of lotteries whose utilities grew fast enough that Sum{i>=0: pi U(Mi)} would fail to converge.
Personally, I am not convinced that bounded utility is the way to go to avoid Pascal's Mugging, because I see no principled way to choose the bound. The larger you make it, the more Muggings you are vulnerable to, but the smaller you make it, the more low-hanging fruit you will ignore: substantial chances of stupendous rewards.
In one of Eliezer's talks, he makes a point about how bad an existential risk to humanity is. It must be measured not by the number of people who die in it when it happens, but the loss of a potentially enormous future of humanity spreading to the stars. That is the real difference between "only" 1 billion of us dying, and all 7 billion. If you are moved by this argument, you must see a substantial gap between the welfare of 7 billion people and that of however many 10^n you foresee if we avoid these risks. That already gives substantial headroom for Muggings.
The M_i's can themselves be lotteries. The idea is to group events into finite lotteries so that the M_i's are >= N.
There is no principled way to chose utility functions either, yet people seem to be fine with them.
My point is that if one takes the VNM theory seriously as justification for having a utility function, the same logic means it must be bounded.
The VNM axioms are the principled way. That's not to say that it's a way I agree with, but it is a principled way. The axioms are the principles, codifying an idea of what it means for a set of preferences to be rational. Preferences are assumed given, not chosen.
Boundedness does not follow from the VNM axioms. It follows from VNM plus an additional construction of infinite lotteries, plus additional axioms about infinite lotteries such as those we have been discussing. Basically, if utilities are unbounded, then there are St. Petersburg-style infinite lotteries with divergent utilities; if all infinite lotteries are required to have defined utilities, then utilities are bounded.
This is indeed a problem. Either utilities are bounded, or some infinite lotteries have no defined value. When probabilities are given by algorithmic probability, the situation is even worse: if utilities are unbounded then no expected utiilties are defined.
But the problem is not solved by saying, "utilities must be bounded then". Perhaps utilities must be bounded. Perhaps Solomonoff induction is the wrong way to go. Perhaps infinite lotteries should be excluded. (Finitists would go for that one.) Perhaps some more fundamental change to the conceptual structure of rational expectations in the face of uncertainty is called for.
I am not sure I understand. Link?
http://arxiv.org/pdf/0907.5598.pdf
So, it's within the AIXI context and you feed your utility function infinite (!) sequences of "perceptions".
We're not in VNM land any more.
But VNM theory allows for bounded utility functions, so if we are designing an agent why don't design it with a bounded utility function?
It would systematically solve Pascal's Mugging, and more formally, it would prevent the expectations to ever become undefined.
Does it? As far as I know, all it says is that the utility function exists. Maybe it's bounded or maybe not -- VNM does not say.
I don't think it would because the bounds are arbitrary and if you make them wide enough, Pascal's Mugging will still work perfectly well.
VNM main theorem proves that if you have a set of preferences consistent with some requirements, then an utility function exists such that maximizing its expectation satisfies your preferences.
If you are designing an agent ex novo, you can choose a bounded utility function. This restricts the set of allowed preferences, in a way that essentially prevents Pascal's Mugging.
Yes, but if the expected utility for common scenarios is not very far from the bounds, then Pascal's Mugging will not apply.
How does that work? VNM preferences are basically ordering or ranking. What kind of VNM preferences would be disallowed under a bounded utility function?
Are you saying that you can/should set the bounds narrowly? You lose your ability to correctly react to rare events, then -- and black swans are VERY influential.
Only in the deterministic case. If you have uncertainty, this doesn't apply anymore: utility is invariant to positive affine transforms, not to arbitrary monotone transforms.
Any risk-neutral (or risk-seeking) preference in any quantity.
I am not sure I understand. Uncertainty in what? Plus, if you are going beyond the VNM Theorem, what is the utility function we're talking about, anyway?
In the outcome of each action. If the world is deterministic, then all that matters is a preference ranking over outcomes. This is called ordinal utility.
If the outcomes for each action are sampled from some action-dependent probability distribution, then a simple ranking isn't enough to express your preferences. VNM theory allows you to specify a cardinal utility function, which is invariant only up to positive affine transform.
In practice this is needed to model common human preferences like risk-aversion w.r.t. money.
Yes, you need risk tolerance / risk preference as well, but once we have that, aren't we already outside of the VNM universe?