# Academian comments on VNM expected utility theory: uses, abuses, and interpretation - Less Wrong

20 17 April 2010 08:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Sort By: Best

Comment author: 18 April 2010 12:06:34PM *  2 points [-]

Indeed, the question is whether non-Archimedean values should be normatively outlawed for rational agents.

Do you think they should? How about for example, an AI whose primary goal is maximize the brightness of a certain light, with a secondary goal is to adjust its color closer to red. If it is (say, by design) totally unwilling to sacrifice some of the latter for the former, would you say it is necessarily irrational? How about a human with the same preferences?

Comment author: 18 April 2010 05:59:21PM *  5 points [-]

Yeah I think non-Archimedian values are normatively irrational, i.e. make an AI less effective as an optimizer. The obvious problem is that when A is "infinitely preferable" to B, then even A's most half-baked subgoal is infinitely preferable to B. Even if I gave the AI a wrapped box with B inside, it wouldn't dedicate any resources to opening it -- better to just flap its arms so that the butterfly effect will increase the chance of A by 1/googol.

Comment author: 18 April 2010 06:48:31PM *  1 point [-]

better to just flap its arms so that the butterfly effect will increase the chance of A by 1/googol

Heh, yeah that's roughly how I feel when noting Archimedeanity of my values. But then I wonder... maybe I wouldn't "flap my arms just so" to increase P(A) because I'm running on hostile hardware that makes my belief probabilities coarse-grained.... i.e. maybe I'm forced to treat 1/googol like 0, and open the box with B in it. I certainly feel that way.

Such reflection leads me to think that humans aren't precise enough for the difference between VNM utility and Hausner utility to really manifest decisively. When would a human really be convinced enough that EUBig(X) precisely equals EUBig(Y), so to start optimizing EUSmall? It seems like the difference between VNM and Hausner utility only happens in a measure-0 class of scenarios that humans couldn't practically detect anyway. ETA: except maybe when there's a time limit...

This is actually one reason I posted on Hausner utility: if you like it, then note it's 0% likely to give you different answers from VNM utility, and then just use VNM because you're not precise enough to know the difference :)

Comment author: 18 April 2010 09:33:38PM *  6 points [-]

Is there any use for introducing the concept of "noise terms" in lexicographical expected value?

Here's an illustration of what I mean by "noise terms":

Suppose that you have a warm cup of coffee sitting in a room. How long will it take before the coffee is the same temperature as the room? Well, the rate of heat transfer between two objects is proportional to the temperature difference between them. That means that the temperature difference between the coffee and the room undergoes exponential decay, and exponential decay never actually reaches its final value. It'll take infinitely long for the coffee to reach the same temperature as the room!

Except that the real world doesn't quite work that way. If you try to measure temperature precisely enough, you'll start finding that your thermometer readings are fluctuating. All those "random" molecular motions create noise. You can't measure the difference between 100 degrees and 100 + epsilon degrees as long as epsilon is less than the noise level of the system. So the coffee really does reach room temperature, because the difference becomes so small that it disappears into the noise.

If the parameters that I use to calculate expected value are noisy, then as long as the difference between EUBig(X) and EUBig(Y) is small enough - below the noise level of the system - I can't tell if it's positive or negative, so I can't know if I should prefer X to Y. As with the coffee in the room, the difference vanishes into the noise. So I'll resort to the tiebreaker, and optimize EUSmall instead.

Does this make any sense?

Comment author: 18 April 2010 10:01:14PM *  0 points [-]

This certainly has an intuitive appeal. Where would the noise be? In the outcome values, or the probabilities, or both?

To have an effect, the noise would have to something that you somehow know can't be eliminated by allocating more thinking/computing time (perhaps you're on a time limit) or resources, else the rational thing to do would be to just "'think harder" to make sure you're not sacrificing your EUBig...

Is that knowledge plausible?

Comment author: 21 April 2010 09:17:50PM 0 points [-]

Good stuff!

Comment author: 18 April 2010 09:04:00PM 0 points [-]

It's not really an issue of impracticality. It's just that, any time you have a higher-level class of utility, the lower-level classes of utility stop being relevant to your decisions. No matter how precise the algorithm is. That's why I say it's extra complexity with no optimization benefit. Since the extra structure doesn't even map better to my intuition about preference, I just Occam-shave it away.

Comment author: 18 April 2010 09:43:13PM *  0 points [-]

any time you have a higher-level class of utility, the lower-level classes of utility stop being relevant to your decisions

Wait... certainly, if you lexicographically value (brightness, redness) of a light, and somehow manage to be in a scenario where you can't make the light brighter, and somehow manage to know that, then the redness value becomes relevant.

What I mean is that the environment itself makes such precise situations rare (a non-practical issue), and an imprecise algorithm makes it hard to detect when, if ever, they occur (a practical issue).

Comment author: 19 April 2010 06:22:23PM 0 points [-]

I wasn't claiming that they should be normatively outlawed, just that in practice in human beings, they lead to logical inconsistency. On the other hand, in a perfect AI, they wouldn't necessarily lead to inconsistency, but the less important goal would be completely ignored, as Liron says, and therefore effectively you would still have Archimedean values.

Comment author: 19 April 2010 02:02:10AM *  0 points [-]

Secondary preferences may as well not exist. The only time they have an effect on behavior is in the case of two or more possible optimal actions with identical expected utilities in terms of the primary good. How likely is that?

ETA: Douglas Knight said it first.

Comment author: 19 April 2010 06:14:25AM *  0 points [-]

Yes (see above). It seems well agreed upon; I think I'll ETA a note about this to the post proper.

ETA: "noise" in your expected utility calculations might constitute positive-likelihood occasions to treat two expectations as equal...