Comment Permalink

Is there any use for introducing the concept of "noise terms" in lexicographical expected value?

Here's an illustration of what I mean by "noise terms":

Suppose that you have a warm cup of coffee sitting in a room. How long will it take before the coffee is the same temperature as the room? Well, the rate of heat transfer between two objects is proportional to the temperature difference between them. That means that the temperature difference between the coffee and the room undergoes exponential decay, and exponential decay never actually reaches its final value. It'll take infinitely long for the coffee to reach the same temperature as the room!

Except that the real world doesn't quite work that way. If you try to measure temperature precisely enough, you'll start finding that your thermometer readings are fluctuating. All those "random" molecular motions create noise. You can't measure the difference between 100 degrees and 100 + epsilon degrees as long as epsilon is less than the noise level of the system. So the coffee really does reach room temperature, because the difference becomes so small that it disappears into the noise.

If the parameters that I use to calculate expected value are noisy, then as long as the difference between EUBig(X) and EUBig(Y) is small enough - below the noise level of the system - I can't tell if it's positive or negative, so I can't know if I should prefer X to Y. As with the coffee in the room, the difference vanishes into the noise. So I'll resort to the tiebreaker, and optimize EUSmall instead.

Does this make any sense?

[anonymous]15y00

I just added a paragraph about this. Good stuff!

0Academian15y

This certainly has an intuitive appeal. Where would the noise be? In the outcome values, or the probabilities, or both? To have an effect, the noise would have to something that you somehow know can't be eliminated by allocating more thinking/computing time (perhaps you're on a time limit) or resources, else the rational thing to do would be to just "'think harder" to make sure you're not sacrificing your EUBig... Is that knowledge plausible?

See in context

36 VNM expected utility theory: uses, abuses, and interpretation

by Academian

17th Apr 2010

12 min read

36

When interpreted convservatively, the von Neumann-Morgenstern rationality axioms and utility theorem are an indispensible tool for the normative study of rationality, deserving of many thought experiments and attentive decision theory. It's one more reason I'm glad to be born after the 1940s. Yet there is apprehension about its validity, aside from merely confusing it with Bentham utilitarianism (as highlighted by Matt Simpson). I want to describe not only what VNM utility is really meant for, but a contextual reinterpretation of its meaning, so that it may hopefully be used more frequently, confidently, and appropriately.

Preliminary discussion and precautions
Sharing decision utility is sharing power, not welfare
Contextual Strength (CS) of preferences, and VNM-preference as "strong" preference
Hausner (lexicographic) decision utility
The independence axiom isn't bad either
Application to earlier LessWrong discussions of utility

1. Preliminary discussion and precautions

The idea of John von Neumann and Oskar Mogernstern is that, if you behave a certain way, then it turns out you're maximizing the expected value of a particular function. Very cool! And their description of "a certain way" is very compelling: a list of four, reasonable-seeming axioms. If you haven't already, check out the Von Neumann-Morgenstern utility theorem, a mathematical result which makes their claim rigorous, and true.

VNM utility is a decision utility, in that it aims to characterize the decision-making of a rational agent. One great feature is that it implicitly accounts for risk aversion: not risking $100 for a 10% chance to win $1000 and 90% chance to win $0 just means that for you, utility($100) > 10%utility($1000) + 90%utility($0).

But as the Wikipedia article explains nicely, VNM utility is:

not designed to predict the behavior of "irrational" individuals (like real people in a real economy);
not designed to characterize well-being, but to characterize decisions;
not designed to measure the value of items, but the value of outcomes;
only defined up to a scalar multiple and additive constant (acting with utility function U(X) is the same as acting with a·U(X)+b, if a>0);
not designed to be added up or compared between a number of individuals;
not something that can be "sacrificed" in favor of others in a meaningful way.

[ETA] Additionally, in the VNM theorem the probabilities are understood to be known to the agent as they are presented, and to come from a source of randomness whose outcomes are not significant to the agent. Without these assumptions, its proof doesn't work.

Because of (4), one often considers marginal utilities of the form U(X)-U(Y), to cancel the ambiguity in the additive constant b. This is totally legitimate, and faithful to the mathematical conception of VNM utility.

Because of (5), people often "normalize" VNM utility to eliminate ambiguity in both constants, so that utilities are unique numbers that can be added accross multiple agents. One way is to declare that every person in some situation values $1 at 1 utilon (a fictional unit of measure of utility), and $0 at 0. I think a more meaningful and applicable normalization is to fix mean and variance with respect to certain outcomes (next section).

Because of (6), characterizing the altruism of a VNM-rational agent by how he sacrifices his own VNM utility is the wrong approach. Indeed, such a sacrifice is a contradiction. Kahneman suggests¹, and I agree, that something else should be added or substracted to determine the total, comparative, or average well-being of individuals. I'd call it "welfare", to avoid confusing it with VNM utility. Kahneman calls it E-utility, for "experienced utility", a connotation I'll avoid. Intuitively, this is certainly something you could sacrifice for others, or have more of compared to others. True, a given person's VNM utility is likely highly correlated with her personal "welfare", but I wouldn't consider it an accurate approximation.

So if not collective welfare, then what could cross-agent comparisons or sums of VNM utilities indicate? Well, they're meant to characterize decisions, so one meaningful application is to collective decision-making:

2. Sharing decision utility is sharing power, not welfare

Suppose decisions are to be made by or on behalf of a group. The decision could equally be about the welfare of group members, or something else. E.g.,

How much vacation each member gets, or
Which charity the group should invest its funds in.

Say each member expresses a VNM utility value—a decision utility—for each outcome, and the decision is made to maximize the total. Over time, mandating or adjusting each member's expressed VNM utilities to have a given mean and variance could ensure that no one person dominates all the decisions by shouting giant numbers all the time. Incidentally, this is a way of normalizing their utilities: it will eliminate ambiguity in the constants ''a'' and ''b'' in (4) of section 1, which is exactly what we need for cross-agent comparisons and sums to make sense.

Without thought as to whether this is a good system, the two decision examples illustrate how allotment of normalized VNM utility signifies sharing power in a collective decision, rather than sharing well-being. As such, the latter is better described by other metrics, in my opinion and in Kahneman's.

3. Contextual strength (CS) of preferences, and VNM-preference as "strong" preference

As a normative thory, I think VNM utility's biggest shortcomming is in its Archimedian (or "Continuity") axiom, which as we'll see, actually isn't very limiting. In its harshest interpretation, it says that if you won't sacrifice a small chance at X in order to get Y over Z, then you're not allowed to prefer Y over Z. For example, if you prefer green socks over red socks, then you must be willing to sacrifice some small, real probability of fulfilling immortality to favor that outcome. I wouldn't say this is necessary to be considered rational. Eliezer has noted implicitly in this post (excerpt below) that he also has a problem with the Archimedean requirement.

I think this can be fixed directly with reinterpretation. For a given context C of possible outcomes, let's intuitively define a "strong preference" in that context to be one which is comparable in some non-zero ratio to the strongest preferences in the context. For example, other things being equal, you might consistently prefer green socks to red socks, but this may be completely undetectable on a scale that includes immortal hapiness, making it not a "strong preference" in that context. You might think of the socks as "infinitely less significant", but infinity is confusing. Perhaps less daunting is to think of them as a "strictly secondary concern" (see next section).

I suggest that the four VNM axioms can work more broadly as axioms for strong preference in a given context. That is, we consider VNM-preference and VNM-utility

to be defined only for a given context C of varying possible outcomes, and
to intuitively only indicate those preferences finitely-comparable to the strongest ones in the given context.

Then VNM-indifference, which they denote by equality, would simply mean a lack of strong preference in the given context, i.e. not caring enough to sacrifice likelihoods of important things. This is a Contextual Strength (CS) interpretation of VNM utility theory: in bigger contexts, VNM-preference indicates stronger preferences and weaker indifferences.

(CS) Henceforth, I explicitly distinguish the terms VNM-preference and VNM-indifference as those axiomatized by VNM, interpreted as above.

4. Hausner (lexicographic) decision utility

[ETA] To see the broad applicability of VNM utility, let's examine the flexibility of a theory without the Archimedean axiom, and see that they differ only mildly in result:

In the socks vs. immortality example, we could suppose that context "Big" includes such possible outcomes as immortal happiness, human extinction, getting socks, and ice-cream, and context "Small" includes only getting socks and ice-cream. You could have two VNM-like utility functions: U_Small for evaluating gambles in the Small context, and U_Big for the Big context. You could act to maximize EU_Big whenever possible (EU=expected utility), and when two gambles have the same EU_Big, you could default to choosing between them by their EU_Small values. This is essentially acting to maximize the pair (EU_Big, EU_Small), ordered lexicographically, meaning that a difference in the former value EU_Big trumps a difference in the latter value. We thus have a sensible numerical way to treat EU_Big as "infinitely more valuable" without really involving infinities in the calculations; there is no need for that interpretation if you don't like it, though.

Since we have the VNM axioms to imply when someone is maximizing one expectation value, you might ask, can we give some nice weaker axioms under which someone is maximizing a lexicographic tuple of expectations?

Hearteningly, this has been taken care of, too. By weakening—indeed, effectively eliminating— the Archimedean axiom, Melvin Hausner² developed this theory in 1952 for Rand Corporation, and Peter Fishburn³ provides a nice exposition of Hausner's axioms. So now we have Hausner-rational agents maximizing Hausner utility.

[ETA] But the difference between Hausner and VNM utility comes into effect only in the rare event when you know you can't distinguish EU_Big values, otherwise the Hausner-rational behavior is to "keep thinking" to make sure you're not sacrificing EU_Big. The most plausible scenario I can imagine where this might actually happen to a human is when making a decision on a precisely known time limit, like say sniping on one of two simultaneous ebay auctions for socks. CronoDAS might say the time limit creates "noise in your expectations". If the time runs out and you have failed to distinguish which sock color results in higher chances of immortality or other EU_Big concerns, then I'd say it wouldn't be irrational to make the choice according to some secondary utility EU_Small that any detectable difference in EU_Big would otherwise trump.

Moreover, it turns out³ that the primary, i.e. most dominant, function in the Hausner utility tuple behaves almost exactly like VNM utility, and has the same uniqueness property (up to the constants ''a'' and ''b''). So except in rare circumstances, you can just think in terms of VNM utility and get the same answer, and even the rare exceptions involve considerations that are necessarily "unimportant" relative to the context.

Thus, a lot of apparent flexibility in Hausner utility theory might simply demonstrate that VNM utility is more applicable to you than it fist appeared. This situation favors the (CS) interpretation: even when the Archimedean axiom isn't quite satisfied, we can use VNM utility liberally as indicating "strong" preferences in a given context.

5. The independence axiom isn't so bad

"A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom." (Wikipedia) But I think the independence axiom (which Hausner also assumes) is a non-issue if we're talking about "strong preferences". The following, in various forms, is what seems to be the best argument against it:

Suppose a parent has no VNM preference between S: her son or her daughter gets a free car, and D: her daughter gets it. In the original VNM formulation, this is written "S=D". She is also presented with a third option, F=.5S+.5D. Descriptively, a fair coin would be flipped, and her son or daughter gets a car accordingly.

By writing S=.5S+.5S and D=.5D+.5D, the original independence axiom says that S=D implies S=F=D, so she must be VNM-indfferent between F and the others. However, a desire for "fair chances" might result in preferring F, which we might want to allow as "rational".

[ETA] I think the most natural fix within the VNM theory is to just say S' and D' are the events "car is awarded so son/daughter based on a coin toss", which are slightly better than S and D themselves, and that F is really 0.5S' + 0.5D'. Unfortunately, such modifications undermine the applicability of the VNM theorem, which implicitly assumes that the source of probabilities itself is insignificant to the outcomes for the agent. Luckily, Bolker⁴ has divised an axiomatic theory whose theorems will apply without such assumptions, at the expense of some uniqueness results. I'll have another occasion to post on this later.

Anyway, under the (CS) interpretation, the requirement "S=F=D" just means the parent lacks a VNM-preference, i.e. a strong preference, so it's not too big of a problem. Assuming she's VNM-rational just means that, in the implicit context, she is unwilling to make certain probabilitstic sacrifices to favor F over S and D.

If the context is Big and includes something like death, the VNM-indifference "S=D" is a weak claim: it might just indicate an unwillingness to increase risk of things finitely-comparable to death in order to obtain F over S or D. She is still allowed to prefer F if no such sacrifice is involved.
If the context is Small, and say only includes her kids getting cars, then "S=D" is a strong claim: it indicates an unwillingless to risk her kids not getting cars to favor S over D in a gamble. Then she can still prefer F, but she couldn't prefer F'=.49S+.49D+.02(no car) over S or D, since it would contradict what "S=D" means in terms of car-sacrifice. I think that's reasonable, since if she simply flips a mental coin and gives her son the car, she can prefer to favor her daughter in later circumstances.

You might say VNM tells you to "Be the fairness that you want to see in the world."

6. Application to earlier other LessWrong discussions of utility

This contextual strength interpretation of VNM utility is directly relevant to resolving Eliezer's point linked above:

"... The utility function is not up for grabs. I love life without limit or upper bound: There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever."

This could just indicate that Eliezer ranks immortality on a scale that trumps finite lifespan preferences, a-la-Hausner utility theory. In a context of differing positive likelihoods of immortality, these other factors are not strong enough to constitute VNM-preferences.

As well, Stuart Armstrong has written a thoughtful article "Extreme risks: when not to use expected utility", and argues against Independence. I'd like to recast his ideas context-relatively, which I think alleviates the difficulty:

In his paragraph 5, he considers various existential disasters. In my view, this is a case for a "Big" context utility function, not a case against independence. If you were gambling only between eistential distasters, then you have might have an "existential-context utility function", U_Existential. For example, would you prefer

90%(extinction by nuclear war) + 10%(nothing), or
60%(extinction by nuclear war) + 30%(extinction by asteroids) + 10%(nothing)?

If you prefer the latter enough to make some comparable sacrifice in the «nothing» term, contextual VNM just says you assign a higher U_Existential to «extinction by asteroids» than to «extinction by nuclear war».⁵ There's no need to be freaked out by assigning finite numbers here, since for example Hausner would allow the value of U_Existential to completely trump the value of U_Everyday if you started worrying about socks or ice cream. You could be both extremely risk averse regarding existential outcomes, and absolutely unwilling to gamble with them for more trivial gains.

In his paragraph 6, Stuart talks about giving out (necessarily normalized) VNM utility to people, which I described in section 2 as a model for sharing power rather than well-being. I think he gives a good argument against blindly maximizing the total normalized VNM utility of a collective in a one-shot decision:

"...imagine having to choose between a project that gave one util to each person on the planet, and one that handed slightly over twelve billion utils to a randomly chosen human and took away one util from everyone else. If there were trillions of such projects, then it wouldn’t matter what option you chose. But if you only had one shot, it would be peculiar to argue that there are no rational grounds to prefer one over the other, simply because the trillion-iterated versions are identical."

(Indeed, practically, the mean and variance normalization I described doesn't apply to provide the same "fairness" in a one-shot deal.)

I'd call the latter of Stuart's projects an unfair distribution of power in a collective decision process, something you might personally assign a low VNM utility to, and therefore avoid. Thus I wouldn't consider it an argument not to use expected utility, but an argument not to blindly favor total normalized VNM utility of a population in your own decision utility function. The same argument—Parfit's Repugnant Conclusion—is made against total normalized welfare.

The expected utility model of rationality is alive and normatively kicking, and is highly adaptable to modelling very weak assumptions of rationality. I hope this post can serve to marginally persuade others in that direction.

References, notes, and further reading:

¹ Kahneman, Wakker and Sarin, 1997, Back to Bentham? Explorations of experienced utility, The quarterly journal of economics.

² Hausner, 1952, Multidimensional utilities, Rand Corporation.

³ Fishburn, 1971, A Study of Lexicographic Expected Utility, Management Science.

⁴ Bolker, 1967, A simultaneous axiomatization of utility and probability, Philosophy of Science Association.

⁵ As wedrifid pointed out, you might instead just prefer uncertainty in your impending doom. Just as in section 5, neither VNM nor Hausner can model this usefully (i.e. in way that allows calculating utilities), though I don't consider this much of a limitation. In fact, I'd consider it a normative step backward to admit "rational" agents who actually prefer uncertainty in itself.

VNM TheoremUtility Functions

Personal Blog

36

Mentioned in

50Why you must maximize expected utility

41Voting is like donating thousands of dollars to charity

26What Should AI Owe To Us? Accountable and Aligned AI Systems via Contractualist AI Alignment

18Interpersonal and intrapersonal utility comparisons

14Reinforcement, Preference and Utility

Load More (5/6)

New Comment

51 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:17 PM

[-]Unknowns15y50

There is still a serious problem with this: you might call it the dust speck-torture problem. If you have a preference regarding socks, and another regarding immortality, and you say that your preference for immortality is infinitely more important than your preference regarding socks, then you are not allowed to find some third term which is comparable both with the socks and with immortality, nor even any finite sequence of terms which will connect them. But in practice this would not apply to any human being; there will always be such a sequence of terms, and thus the person would have to compare socks and immortality in order to remain rational.

[-]Academian15y30

Indeed, the question is whether non-Archimedean values should be normatively outlawed for rational agents.

Do you think they should? How about for example, an AI whose primary goal is maximize the brightness of a certain light, with a secondary goal is to adjust its color closer to red. If it is (say, by design) totally unwilling to sacrifice some of the latter for the former, would you say it is necessarily irrational? How about a human with the same preferences?

[-]Liron15y60

Yeah I think non-Archimedian values are normatively irrational, i.e. make an AI less effective as an optimizer. The obvious problem is that when A is "infinitely preferable" to B, then even A's most half-baked subgoal is infinitely preferable to B. Even if I gave the AI a wrapped box with B inside, it wouldn't dedicate any resources to opening it -- better to just flap its arms so that the butterfly effect will increase the chance of A by 1/googol.

[-]Academian15y20

better to just flap its arms so that the butterfly effect will increase the chance of A by 1/googol

Heh, yeah that's roughly how I feel when noting Archimedeanity of my values. But then I wonder... maybe I wouldn't "flap my arms just so" to increase P(A) because I'm running on hostile hardware that makes my belief probabilities coarse-grained.... i.e. maybe I'm forced to treat 1/googol like 0, and open the box with B in it. I certainly feel that way.

Such reflection leads me to think that humans aren't precise enough for the difference between VNM utility and Hausner utility to really manifest decisively. When would a human really be convinced enough that EUBig(X) precisely equals EUBig(Y), so to start optimizing EUSmall? It seems like the difference between VNM and Hausner utility only happens in a measure-0 class of scenarios that humans couldn't practically detect anyway. ETA: except maybe when there's a time limit...

This is actually one reason I posted on Hausner utility: if you like it, then note it's 0% likely to give you different answers from VNM utility, and then just use VNM because you're not precise enough to know the difference :)

[-]CronoDAS15y80

Is there any use for introducing the concept of "noise terms" in lexicographical expected value?

Here's an illustration of what I mean by "noise terms":

Does this make any sense?

[-][anonymous]15y00

I just added a paragraph about this. Good stuff!

[-]Academian15y00

This certainly has an intuitive appeal. Where would the noise be? In the outcome values, or the probabilities, or both?

To have an effect, the noise would have to something that you somehow know can't be eliminated by allocating more thinking/computing time (perhaps you're on a time limit) or resources, else the rational thing to do would be to just "'think harder" to make sure you're not sacrificing your EUBig...

Is that knowledge plausible?

[-]Academian15y00

Okay, I just added a paragraph about this. It seems to me that time limits are the biggest noise factor for humans.

Good stuff!

[-]Liron15y00

It's not really an issue of impracticality. It's just that, any time you have a higher-level class of utility, the lower-level classes of utility stop being relevant to your decisions. No matter how precise the algorithm is. That's why I say it's extra complexity with no optimization benefit. Since the extra structure doesn't even map better to my intuition about preference, I just Occam-shave it away.

[-]Academian15y00

any time you have a higher-level class of utility, the lower-level classes of utility stop being relevant to your decisions

Wait... certainly, if you lexicographically value (brightness, redness) of a light, and somehow manage to be in a scenario where you can't make the light brighter, and somehow manage to know that, then the redness value becomes relevant.

What I mean is that the environment itself makes such precise situations rare (a non-practical issue), and an imprecise algorithm makes it hard to detect when, if ever, they occur (a practical issue).

[-]Unknowns15y00

I wasn't claiming that they should be normatively outlawed, just that in practice in human beings, they lead to logical inconsistency. On the other hand, in a perfect AI, they wouldn't necessarily lead to inconsistency, but the less important goal would be completely ignored, as Liron says, and therefore effectively you would still have Archimedean values.

[-]Matt_Simpson15y00

Secondary preferences may as well not exist. The only time they have an effect on behavior is in the case of two or more possible optimal actions with identical expected utilities in terms of the primary good. How likely is that?

ETA: Douglas Knight said it first.

[-]Academian15y00

Yes (see above). It seems well agreed upon; I think I'll ETA a note about this to the post proper.

ETA: "noise" in your expected utility calculations might constitute positive-likelihood occasions to treat two expectations as equal...

[-]Douglas_Knight15y30

Yes, if you drop continuity, all you get is lexicographic preferences, which isn't much of a change. But situations where you want to break ties don't come up much. There is usually the opportunity to measure more closely the first-order goal, rather than to try to control the tie-breaker.

[-]Liron15y30

I don't see that you've offered a better alternative to VNM utility.

Your examples assert that VNM does a bad job of capturing subtleties in the structure of human preference. But they can all be fixed by using the appropriate outcome-space as your utility function's domain.

For example, if I prefer to decide which child gets a car with a fair coin flip, then I can represent my utility function like this:

U(son gets car after coin flip) = 10.1  
U(daughter gets car after coin flip) = 10.1  
U(son gets car after my arbitrary decision) = 10  
U(daughter gets car after my arbitrary decision) = 10

To me this seems like an elegant enough way to capture my value of fairness. And this kind of VNM formulation is so precise that I think it is useful for defining exactly how I value fairness.

I think the alternate utility theories introduce complexity without improving decision theory.

[-]Academian15y20

Unfortunately, what you describe deviates from the VNM model, which does not allow utilities that depend on the source of "objective randomness" in the lotteries; in this case, the coin. If we allow this dependence, it leads to difficulty in defining what probability is, or at least characterizing it behaviorally, and in proving the VNM theorem itself. This is all explained Bolker (1967) - A simultaneous axiomatization of utility and probability, which is probably my favorite paper on utility theory.

The key to building an axiomatic theory where your approach makes sense, due to Bolker, is to axiomatize expectation itself. I originally planned to post on Bolker's theory, but unfortunately, the axioms themselves are a bit abstract, so I decided I should at least start with VNM utility. But Bolker's axioms allow one to work in the exactly the way you describe, which is why I like it so much.

You didn't mention what I think is most important, though, and why I wrote this post: what do you think of the issues regarding the Archimedean axiom?

[-]Liron15y60

When it comes to biting VNM axioms, the Archimedian/continuity axiom is a non-bullet. Why do you have to insist that 1/3^^^3 human lives is an insultingly high price for green socks? I think it's a bargain. Like I said, this infinitesimal business adds complexity with no decision theory payoff.

I don't see why the son-daughter-car formulation I described isn't good enough as is. The domain of my utility function is the set of spans of future time, and the function works by analyzing the causal relationship between my brain and the person who gets the car.

And I don't see why "objective randomness" or anything else needs to come into the picture. I already have a structure that captures what I intuitively mean by "preference" for this example, and an algorithm that makes good decisions accordingly.

[-]Academian15y30

Why do you have to insist that 1/3^^^3 human lives is an insultingly high price for green socks?

At no point have I insisted that, or anything analogous. But that's not the question at hand. I haven't found any of my values to be non-Archimedean. The normative question is whether they should be outlawed for rational agents. From the post:

I'd say you're allowed to choose the green socks over the red socks every time, while exhibiting no willingness to sacrifice Big context expected utility for it, and still be considered "rational".

What's your take? Do you think it should be normatively illegal to have non-Archimedian values (like say, a pair of Hausner utilities) and be considered rational? Please share you thoughts on this thread if you're interested.

I don't see why the son-daughter-car formulation I described isn't good enough as is

As far as I'm concerned, it is good enough. So then the challenge is to provide reasonable assumptions which

allow the kind of analysis you describe, and
still manage to imply mathematical theorems comparable in strength to those of VNM.

VNM utility theory isn't just an intuitive model. It's special: it's actually a consistent mathematical theory, with theorems. And the task of weakening assumptions while maintaining strong theorems is formidable. Luckily, Bolker has done this, by axiomatizing expectation itself, so when we want rigorous theorems to fall back on while reasoning as you describe, they're to be found.

And I don't see why "objective randomness" or anything else needs to come into the picture

AFAIC, it doesn't. But whether we like it or not, the VNM theory needs the source of randomness to not be a source of utility in order for the proof of the VNM utility theorem to work. I find this unsatisfactory, as you seem to. Luckily, Bolker's theory doesn't require this, which is great. Instead, Bolker pays a different price:

his utility is no longer unique up to two constants, as it is VNM.
probabilities themselves become behaviorally ambiguous.

This doesn't bother me much, nor you probably; I consider it a price easily worth paying to pass from VNM to Bolker.

I admit this comment is not as in-depth an explanation of some concepts as I'd like; if I find it ties together with enough interesting topics, and I think it will, then I'll write a top level post better explaining this stuff.

[-]Liron15y10

But I did upvote this post, because I think it's a good discussion to have.

[-]linkhyrule511y20

...

I'm just going to say "tiered utilities" here, so that the next person who thinks like me and tries to see if anyone's written anything about (what turns out to be) Hausner utility can find this post easily.

[-]thomblake15y00

Perhaps I'm missing something, but it seems to me this doesn't cover risk-aversion properly.

One great feature is that it implicitly accounts for risk aversion: not risking $100 for a 10% chance to win $1000 and 90% chance to win $0 just means that for you, utility($100) > 10%utility($1000) + 90%utility($0).

Suppose for me, utility($100) = 1, and utility($1000) = 100, and utility($0) = 0. Then, utility($100) < 10%utility($1000) + 90%utility($0); (1 < 10). Now suppose I am extremely risk-averse; I prefer to never wager any money I actually have and will practically always take a certain $100 over any uncertain $1000. It does not seem this configuration is impossible in an agent, but is not supported by your model of risk-aversion.

Does this merely mean that this type of risk-aversion is considered irrational and therefore not covered in the VNM model?

[-]pengvado15y00

Does this merely mean that this type of risk-aversion is considered irrational and therefore not covered in the VNM model?

Yes. If you are risk-averse (or loss-averse) in terms of marginal changes in your money, then you're not optimizing any consistent function of your total amount of money.

[-]Matt_Simpson15y00

Yet there is apprehension about its applicability, aside from merely confusing it with Bentham utilitarianism (as highlighted by Matt Simpson).

I wasn't (directly) talking about utilitarianism in that post. I was talking about using utility functions to model two different things: behavior and preferences. Utilitarianism is a specific example of the latter.

[-]wedrifid15y00

For example, would you prefer

90%(extinction by nuclear war) + 10%(nothing), or

60%(extinction by nuclear war) + 30%(extinction by asteroids) + 10%(nothing)?

If you prefer the latter enough to make some comparable sacrifice in the «nothing» term, and you're rational, I'd say say you assign a higher UExistential to «extinction by asteroids» than to «extinction by nuclear war» (hopefully both negative numbers).

Under the system you describe it is possible to weight each of the below scenarios equally:

0.6(extinction by nuclear war) + (0.3 + 2e)(extinction by asteroids) + (0.1 - 2e)(nothing)?
0.01(extinction by nuclear war) + (0.89 + 1e)(extinction by asteroids) + (0.1 - e)(nothing)?
0.9(extinction by nuclear war) + 0.1(nothing)

You have allowed for agents to assign different weights for different probabilities of certain stuff happening. This does more than allow for risk-aversion. It allows risk-seeking and arbitrary risk appreciation such as preferring a any split of two kinds of existential catastrophe to 90% of either. You can conclude nothing about whether you prefer asteroids to nuclear war based on preference between the scenarios that either of us just listed. You can not shut up and multiply (except when the agents whims it as such).

[-]Academian15y00

In the VNM model, the probabilities are assumed to be from a source which is indifferent to the agent, and numerically agreed upon. That's how the proof of the theorem works. We already discussed this above, but thanks for writing out an example to explicitly demonstrate.

[-]wedrifid15y00

The discussion you linked to does not appear to be related to my comment. Unless for some reason you have rejected parts of your original post this claim is mistaken:

If you prefer the latter enough to make some comparable sacrifice in the «nothing» term, and you're rational, I'd say say you assign a higher UExistential to «extinction by asteroids» than to «extinction by nuclear war» (hopefully both negative numbers).

[-]Academian15y00

When I say

In the VNM model, the probabilities are assumed to be from a source which is indifferent to the agent, and numerically agreed upon

I mean that it would consider you modifying the probabilities as simply presenting a different problem.

Maybe you're asserting that, in principle, the assumption that we be sure of the agent's estimation of the probabilities can't be met. I would agree.

[-]wedrifid15y00

I mean that it would consider you modifying the probabilities as simply presenting a different problem.

I didn't modify the probabilities. Unless you consider making the 'e' that you expressed in words explicit or converting percentage to decimal modification 'modificaiton'. I also added a third scenario to demonstrate a compatible alternative.

I have concluded that the opening post contains a trivial error that is unlikely to be corrected. The worst place for errors is in an official looking context with otherwise high quality content, so I have reversed my upvote.

[-]Academian15y00

Unless you consider making the 'e' that you expressed in words explicit ...

Ah! I see what you mean, and now agree that you didn't modify the scenario. So I can better understand, when you say

You can not shut up and multiply (except when the agents whims it as such).

does the "except" clause mean "except when the agent is VNM rational", or some similar assumption?

If your point is that I can't deduce you are VNM-rational from your response to the extinction question alone, then you're definitely right. If I came across to the contrary, I should reword the post.

What I meant to point out in that post paragraph is just that

1) there is no need to be "freaked out" by assigning finite utilities to huge events, because

2) even if you insist on treating them as "infinitely more important" than everyday events, you can just use Hausner utility to break ties in more trivial decisions (where the EUexistential values are equal, or "noisily indistinguishable" due for example to a precise time limit on a decision, after which time you've failed to distinguish existential risks from noise). I consider this an argument in favor of using in-context VNM utility normatively, since it is simpler than Hausner, and differs from it rarely, over relatively unimportant matters in the context.

I want to convince as many people as possible to use VNM utility confidently. I'd value any suggestions you have to rephrase the post to that effect, since (I think) you noticed the above unclarity in the first place.

[-]wedrifid15y10

If your point is that I can't deduce you are VNM-rational from your response to the extinction question (and further questions) alone? Then you're definitely right. If I came across to the contrary, I should reword the post.

Exactly! Perhaps just change "I'd say you assign a higher" to "then maybe you assign a higher". That would be enough to stop people (me) twisting their minds in a knot trying to work out whether there is an oversight or whether they just don't understand the whole VNM thing. I would also add something like "(or perhaps you just value unpredictability in your impending doom!)", which would help the curious mind (or, me again!) to confirm their understanding of now the VNM structure allows for valuing uncertain events.

I think I agree with your point. I am dubious about the whole 'infinitely more important' thing but if VNM handles that in a sane way then the problem of beliefs about infinity rests on the agent and his map and VNM can just handle whatever values and expectations the agent happens to have.

(Thanks for taking the time to take another look at what I was trying to say. It isn't often that I see conversations that go "No -> huh? Actually. -> wtf? No really" resolve themselves to agreement. So my prediction is one I am glad to be mistaken on!)

[-]Academian15y20

Okay, I rewrote the ending, and added a footnote about "valuing uncertainty". Thanks for all the input!

It isn't often that I see conversations that go "No -> huh? Actually. -> wtf? No really" resolve themselves to agreement.

Maybe it helps that we're both more concerned with figuring stuff out than "winning an argument" :)

[-]Tyrrell_McAllister15y00

Possible typo :)

For example, if you prefer green socks over green socks, then you must be willing to sacrifice some small, real probability of fulfilling immortality to favor that outcome.

(Emphasis added.)

[-]Academian15y00

Thanks! Fixed.

[-]JGWeissman15y00

"... The utility function is not up for grabs. I love life without limit or upper bound: There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever."

This could just indicate that Eliezer ranks immortality on a scale that trumps finite lifespan preferences, a-la-Hausner utility theory. In a context of differing positive likelihoods of immortality, these other factors are not strong enough to constitute VNM-preferences.

Knowing that Eliezer is an "infinite set atheist" and uses infinity as the limit of a sequence of finite values, I would interpret his statement as:

For every finite amount of life lived N, there exists a finite amount of life lived M, such that I would prefer an 0.0001% chance of living a googolplex years and an 80% chance of living M years to a 80.0001% probability of living N years.

You can replace 0.0001% with epsilon, giving:

For every probability epsilon > 0 (and epsilon < 20%) every finite amount of life lived N, there exists a finite amount of life lived M, such that I would prefer an epsilon chance of living a googolplex years and an 80% chance of living M years to a 80% + epsilon probability of living N years.

This does not violate continuity, because M depends on epsilon.

[-]Academian15y00

This is not necessary to give meaning to immortality without reference to infinite sets. Immortality is the non-existence of a finite time of death. The statement can be regarded as valuing that scenario directly, without mentioning M or epsilon.

[-]JGWeissman15y00

Immortality is the non-existence of a finite time of death.

This depends on the assumption that there is an infinite amount of time. You have only hidden, not eliminated, your reference to an infinite set.

[-]Academian15y10

Certainly not. There are no infinite sets in first-order Peano arithmetic, and no largest number, either.

[-]JGWeissman15y00

There are no infinite sets in first-order Peano arithmetic

Perhaps you mean that first-order Peano arithmetic cannot prove that the set of natural numbers is infinite.

[-]Sniffnoy15y40

Well, first-order Peano arithmetic doesn't have any notion of "set", much less "infinite"...

[-]JGWeissman15y00

Right, that is my point. If an axiomatic system has no concept of "set" or "infinite", it will not notice if the thing it is describing is in fact an infinite set. But we can use a more complete system, and notice that it is an infinite set and make deductions from it. Which is why first-order Peano arithmetic does not invalidate my claim that Academian's characterization of immortality relies on a hidden reference to an infinite set.

[-]Academian15y00

The standard model of Peano arithmetic is 0={} and S(X)=Xunion{X}; its objects are all finite sets.

I fail to see any meaningful sense in which saying "There does not exist a time T after the present such that I am not alive" has a hidden reference to infinity which is somehow avoided by saying "For any duration N and probability epsilon, there exists a longer duration M such that..."

[-][anonymous]15y00

$\ucap$

[-]JGWeissman15y00

The standard model of Peano arithmetic is 0={} and S(X)=Xunion{X}; its objects are all finite sets.

The set of all objects in this standard model is infinite, even though Peano arithmetic does not refer to this set.

I fail to see any meaningful sense in which ""There does not exist a time T after the present such that I am not alive" has a "hidden reference to infinity" which is somehow avoided by saying "For any duration N and probability epsilon, there exists a longer duration M such that..."

The difference is that in my version, utility is assigned to states of a finite universe. To the extent that there is a reference to infinity, the infinity describes an abstract mathematical object. In your version, utility is assigned to a state of an infinite universe. The infinity describes physical time, it says that there is an infinite amount of time you will be alive.

[-]Academian15y00

The difference is that in my version, utility is assigned to states of a finite universe.

That's true. You prefer to assign infinitely many utilities to infinitely many states of possible finite universes, and I allow assigning one utility to one state of a possibly infinite universe.

I think the reasons for favoring each case are probably clear to us both, so now I vote for not generating further infinities of comments in the recent comments feed :)

[-]Academian15y00

No, I mean that there are no infinite sets in first-order Peano arithmetic. The class "natural number" is not an object in Peano arithmetic.

[-]JGWeissman15y-20

First-order Peano arithmetic does not explicitly refer to sets, infinite sets, or natural numbers, but it describes them.

[-]Academian15y-20

Correct. Nor does "There is no time of death" refer to a set. You can model it as quantifying over a "set of times" if you like, just as Peano arithmetic can be modeled as quantifying over a "set of numbers", but this does not mean the theory refers to sets, or infinity.

[-][anonymous]15y-20

There are no sets at all in first-order Peano arithmetic--sets are simply not a thing that the theory talks about.

(People talking about first-order Peano arithmetic, however, do talk about sets a lot.)

[-]wedrifid15y00

Unless for some reason time is constructed in a way that it would not end in a way that implies 'death'. Loops, or something that just hasn't occurred to me. Stranger things in physics than that are true.

[-]JGWeissman15y00

I am not sure how to interpret your comment.

I am claiming that the reason someone might assign infinite utility to infinite life is that it is infinite, and that ways of describing infinite life that don't mention infinity, and rely on unspoken beliefs that something is infinite.

Do you contradict this claim?

[-]wedrifid15y00

Do you contradict this claim?

No. I assert the claim that I made, which is relevant to that comment's parent.

A related background assumption that I don't make is that 'no finite time of death' implies 'infinite life'. There are specifications of the physics of time that I assign non zero probability of being the correct map for which a reasonable specification of 'immortal' relies no infinite duration of time.

Moderation Log