You cannot feed a probability of 1 into a utility function. It makes no sense.

I think "U(Certainty)" was meant to be shorthand for U(feeling of certainty). Otherwise - well said.

Somewhat related: Donald Hoffman on veridical perception.

Hmmm.... I thought the point of your article at http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/ was that the difference between 1 and .99 was indeed much larger than, say, .48 and .49.

Anyway, what if we try this one on for size: let's say you are going to play a hand of Texas Hold 'em and you can choose one of the following three hands (none of them are suited): AK, JT, or 22. If we say that hand X > Y if hand X will win against hand Y more that 50% of the time, then AK > JT > 22 > AK > JT ..... etc. So in this case couldn't one choose rationally and yet still be a "money pump"?

No, because you don't want to switch to a hand that will beat the one you have, you want to switch to one that's more likely to beat your opponent's (unknown, fixed) hand. That's necessarily transitive.

Okay, just one more question, Eliezer: when are you going to sit down and condense your work at Overcoming Bias into a reasonably compact New York Times bestseller?

James Bach, your point and EY's are not incompatible : it is a given that what you care about and give importance to is subjective and irrational, however having chosen what outcomes you care about, your best road to achieving them must be Bayesian.... perhaps. My problem with this whole Bayesian kick is that it reminds me of putting three masts and a full set of square-rigged sails on what is basically a canoe : the masts and sails are the Bayesian edifice, the canoe is our useful knowledge in any given real life situation.

tcpkac - that's what they said to Columbus.

The circular money pump kept bringing M C Escher illustrations to my mind - the never-ending staircase in particular. This post cleared up a lot of what I didn't take in yesterday - thanks for taking the time.

There is no special rational basis for claiming that when lives are at stake, it's especially important to be rational.

James - the reason 'lives at stake' comes up in examples is because the value we place on human life tends to dwarf everything else. Just because that value is enormous, doesn't mean it's uncalculable. Considering lives is the best way to force ourselves to think as economically as we can - more so than money (many people are rich already). It may give us a pang of cold, inhuman logicality to sit down with a piece of paper and a pen to work out the best way to save lives, but that's the game.

Just a quick correction-

Experimental subjects Experimenters tend to defend attack incoherent preferences even when they're really silly strongly held.

;->

I guess what I'd like to know is whether you are a) trying to figure out what people do, or b) trying to predict outcomes and then tell people what to do? Despite my slightly snarky tone, as a curious outsider I really am curious as to which you take to be your goal. Coming from a science-y background, I can totally understand b), but life has shown me plenty of instances of people acing contrary to b)'s predictions.

Maybe the reason we tend to choose bet 2 over bet 1 (before computing the actual expected winnings) is not the higher probability to win, but the smaller sum we can lose (either we expect to lose or we can lose at worst, I'm not sure about that). So the bias here could be more something along the lines of status quo bias or endownment effect than a need for certainty.

I can only speak for myself, but I do not intuitively value certainty/high probability of winning, while I am biased towards avoiding losses.

On second thought, strike my second paragraph (in the 10:08 am comment).

I shouldn't have tinkered with the (well-known?) example I first heard. It's a coke machine on a military base that introduced a price increase by dispensing cokes with a lower probability rather than with a nominal price increase. To the soldiers that live and work there, the machine is equivalent to one with a nominal price increase. The philosopher's question is whether this machine is fair to someone who is just passing through, someone for whom there is no long run.

What I mean to suggest by the example is that the chance deals you offer people do not have a long-run, and your scheme of rational choice is hard to justify without a long-run, no? Can you say something about this, Eliezer?

To the soldiers that live and work there, the machine is equivalent to one with a nominal price increase.

No, it isn't. It's possible that random happenstance could deny some people more cans than others. The set of outcomes only becomes equivalent to raising the price as the number of uses increases to infinity - and that's never assumable in the real world.

It's not fair to the people that live and work there, either.

"Not yet"

So it's going to happen eventually! Yay!

Back on topic, I second Lee's thoughts. My ability to do a simple expected utility calculation would prevent me from ever taking option (1) given a choice between the two, but if I badly needed the $4 for something I might take it. (Initially hard to think of how such a scenario could arise, but $4 might be enough to contact someone who would be willing to bail me out of whatever trouble I was in.)

Dagon made a point about the social importance of guarantees. If a promise is broken, you know you have been cheated. If you are persuaded that there is only a 10% chance of losing your investment and you are unlucky, what do you know?

I doubt that we can over come our biases by focusing on what they are bad for and how they hurt us. We also need to think about what they are good for and why we have them.

Long run? What? Which exactly equivalent random events are you going to experience more than once? And if the events are only really close to equivalent, how do you justify saying that 30 one-time shots at completely different ways of gaining 1 utility unit is a fundamentally different thing than a nearly-exactly-repeated game where you have 30 chances to gain 1 utility unit each time?

There is nothing irrational about choosing 1A over 1B or choosing 2B over 2A. Combining the two into a single scheme or an iterated choice are totally different situations from the original proposition.

Too much research on cognition, especially biases, tends to infer too much from simplified experiments. True, in this case many people slip into a money pump situation easily, but the original proposition does not require that to occur.

Context is king.

This sort of dilemma depends on context. Some may have been cheated in the past, so certainty is valuable to them. Others may need exactly $24,000, and others may need exactly $27,000 for a larger (higher utility) purpose. Others may have different risk tolerance.

You may argue that, given only this decision and no outside influences, a person would be irrational to choose a particular way. Unfortunately, you will never find a reasoning being without context.

This is exactly the Bayesian way. Previous experience defines what is currently rational. Later experience may show the earlier actions to have been imperfect, or unwise to repeat. But to say that we are irrational because we are basing our decision on our own personal context is to deny everything that you have built up to this point.

Context is everything following, "E(X given ". Do not deny the value of it by asserting that, in one specific instance, it mislead us. We may learn from additional data, but it did not mislead us.

I see the importance of context generally overlooked in this discussion, as in most discussion of rationality (encompassing discussion of rational methods.) The elegance and applicability of Bayesian inference is to me undeniable, but I look forward to broader discussion of its application within systems of effective decision-making entailing prediction within a context which is not only uncertain but evolving. In other words, consideration of principles of effective agency where the game itself is inherently uncertain.

In doing so, I think we are driven to re-frame our thinking away from goals to be achieved and onto values to be promoted, away from rules to be followed and onto increasingly coherent principles to be applied, and away from maximizing expected utility and onto maximizing potential synergies within a game that is consistent but inherently incomplete. I see Bayes as necessary but not sufficient for this more encompassing view.

BusinessConsultant: But to say that we are irrational because we are basing our decision on our own personal context is to deny everything that you have built up to this point. Really? If a decision is irrational, it's irrational. You can make allowances for circumstance and still attempt to find the most rational choice. Did you read the whole post? Eliezer is at pains to point out that even given different expected utilities for different amounts of money for different people in different circumstances, there is still a rational way to go about making a decision and there is stilla tendency for humans to make bad decisions because they are too lazy (my words, not his) to think it through, instead trusting their "intuition" because it "feels right."

The point about paying two human lives to flip the switch and then switch it back really drove home the point, Eliezer. Also, a good clarification on consistency. Reading the earlier post, I also thought of the objection that $24,000 could change a destitute person's life by orders of magnitude, whereas $3000 on top of that would not be equivalent to 1/8 more utility... the crucial difference for a starving, sick person is in, say, the first few grand.

But then, as you point out, your preference for the surer chance of less money should remain consistent however the game is stated. Thanks! Very clear...

Also, living in New York and longing for Seattle, I found myself visiting Seattle for Christmas and longing for New York... hmmm. Maybe I just need a taxi to Oakland. :P

"A number of commenters, yesterday, claimed that the preference pattern wasn't irrational because of "the utility of certainty", or something like that. One commenter even wrote U(Certainty) into an expected utility equation."

It was not my intent to claim "the preference pattern wasn't irrational," merely that your algebraic modeling failed to capture what many could initially claim was a salient detail of the original problem. I hope a reread of my original comment will find it pleading, apologetic, limited to the algebraic construction, and sincere.

I should have mentioned that I thought the algebraic modeling was a very elegant way to show that the diminishing marginal utility of money was not at play. If that was its only purpose, then the rest of this is unnecessary, but I think you can use that construction to do more, with a little work.

Here's one possible response to this apparent weakness in the algebraic modeling:

If you can simply assert that Allais's point holds experimentally for arbitrarily increasing values in place of $24k and $27k (which I'm sure you can), then we find this proposed "utility of certainty" (or whatever more appropriate formulation you prefer*) increasing with no upper bound. The notion that we value certainty seems to hold intuitive appeal, and I see nothing wrong with that on its face. But the notion that we value certainty above all else is more starkly implausible (and I would suspect demonstrably untrue: would you really give your life just to become certain of the outcome of a coinflip?).

I was trying to make the argument stronger, not weaker, but I get the impression I've somehow pissed all over it. My apologies.

*I've read your post on Terminal Values three times and haven't yet grokked why I can't feed things like knowledge or certainty into a Utility function. Certainty seems like a "fixed, particular state of the world," it seems like an "outcome," not a "action," and most definitely unlike "1." If the worry is that certainty is an instrumental value, not a terminal value, why couldn't one make the same objection of the $24,000? Money has no inherent value, it is valuable only because it can be spent on things like chocolate pizza. You've since replaced the money with lives, but was the original use of money an error? I suspect not... but then what is the precise problem with U(Certainty)?

I should clarify that, once again, I bring up these objections not to show where you've gone wrong, but to show where I'm having difficulties in understanding. I hope you'll consider these comments a useful guide as to where you might go more slowly in your arguments for the benefit of your readers (like myself) who are a bit dull, and I hope you do not read these comments as combative, or deserving of some kind of excoriating reply.

I'll keep going over the Terminal Values post to see if I can get it to click.

There is a certain U(certainty) in a game, although there might be better ways to express it mathematically. How do you know the person hosting the game isn't lying to you an really operating under the algorithm: 1A. Give him $24,000 because I have no choice. 1B. Tell him he had a chance to win but lost and give nothing.

In the second situation(2A 2B) both options are probabilities and so the player has no choice but to trust the game host.

Also, I am still fuzzy on the whole "money pump" concept. "The naive preference pattern on the Allais Paradox is 1A > 1B and 2B > 2A. Then you will pay me to throw a switch from A to B because you'd rather have a 33% chance of winning $27,000 than a 34% chance of winning $24,000."

Ok, I pay you one penny. You might be tricking me out of one penny(in case you already decided to give me nothing) but I'm willing to take that risk.

"Then a die roll eliminates a chunk of the probability mass. In both cases you had at least a 66% chance of winning nothing. This die roll eliminates that 66%. So now option B is a 33/34 chance of winning $27,000, but option A is a certainty of winning $24,000. Oh, glorious certainty! So you pay me to throw the switch back from B to A."

Yes yes yes, I pay you 1 penny. You now owe me $24,000. What? You want to somehow go back to a 2A 2B situation again? No thanx. I would like to get my money now. Once you promised me money with certainty you cannot inject uncertainty back into the game without breaking the rules.

I'm afraid there might still be some inferential distance to cover Eliezer.

How do the commenters who justify the usual decisions in the face of certainty and uncertainty with respect to gain and loss account for this part of the post?

There are various other games you can also play with certainty effects. For example, if you offer someone a certainty of $400, or an 80% probability of $500 and a 20% probability of $300, they'll usually take the $400. But if you ask people to imagine themselves $500 richer, and ask if they would prefer a certain loss of $100 or a 20% chance of losing $200, they'll usually take the chance of losing $200. Same probability distribution over outcomes, different descriptions, different choices.

Assuming that this experiment has actually been validated, there's hardly a clearer example of obvious bias than a person's decision on the exact same circumstance being determined by whether it's described as certain vs. uncertain gain or certain vs. uncertain loss.

And Eliezer, I have to compliment your writing skills: when faced with people positing a utility of certainty, the first thing that came to my mind was the irrational scale invariance such a concept must have if it fulfills the stated role. But if you'd just stated that, people would have argued to Judgment Day on nuances of the idea, trying to salvage it. Instead, you undercut the counterargument with a concrete reductio ad absurdum, replacing $24,000 with 24,000 lives- which you realized would make your interlocutors uncomfortable about making an incorrect decision for the sake of a state of mind. You seem to have applied a vital principle: we generally change our minds not when a good argument is presented to us, but when it makes us uncomfortable by showing how our existing intuitions conflict.

If and when you publish a book, if the writing is of this quality, I'll recommend it to the heavens.

there's hardly a clearer example of obvious bias than a person's decision on the exact same circumstance being determined by whether it's described as certain vs. uncertain gain or certain vs. uncertain loss.

But it's not the exact same circumstance. You are ignoring the fundamental difference between the two conditions.

But it's not the exact same circumstance. You are ignoring the fundamental difference between the two conditions.

Show us. Use maths.

Ben Jones, and Patrick (orthonormal), if you offer me 400$ I'll say 'yes, thank you'. If you offer me 500$ I'll say 'yes, thank you'. If, from whatever my current position is after you've been so generous, you ask me to choose between "a certain loss of $100 or a 20% chance of losing $200", I'll choose the 20% chance of losing 200$. That's my math, and I accept money orders, wire transfers, or cash....

You come out with the same amount of money, but a different thing happens to get you there. This matters emotionally, even though it shouldn't (or seems like it shouldn't). A utility function can take things other than money into account, you know.

Show us. Use maths.

The distinction already exists in the natural language used to describe the two scenarios.

In one scenario, we are told that a certain amount of money will become ours, but we do not yet possess it. In the other, we consider ourselves to already possess the money and are given opportunities to risk some of it.

Hypothetical money is not treated as equivalent to possessed money. (Well, hypothetical hypothetical vs. possessed hypothetical in the experiment discussed, but you know what I mean.)

This matters emotionally, even though it shouldn't (or seems like it shouldn't).

Hypothetical money is not treated as equivalent to possessed money.

My point exactly. It's perfectly understandable that we've evolved a "bird in the hand/two in the bush" heuristic, because it makes for good decisions in many common contexts; but that doesn't prevent it from leading to bad decisions in other contexts. And we should try to overcome it in situations where the actual outcome is of great value to us.

A utility function can take things other than money into account, you know.

As well it should. But how large should you set the utilities of psychology that make you treat two descriptions of the same set of outcomes differently? Large enough to account for a difference of $100 in expected value? $10,000? 10,000 lives?

At some point, you have to stop relying on that heuristic and do the math if you care about making the right decision.

But how large should you set the utilities of psychology that make you treat two descriptions of the same set of outcomes differently?

As far as I'm concerned, zero; we agree on this. My point was only that it's misleading to say "the same set of outcomes" or "the same circumstance" for the same amount of money; a different thing happens to get to the same monetary endpoint. It's not a difference that I (or my idealized self) care(s) about, though.

Similarly, I think it's misleading to say "choosing 1A and 2B is irrational" without adding the caveat "if utility is solely a function of money, not how you got that money".

"Doing the math" requires that we accept a particular model of utility and value, though. And this is why people are objecting to Eliezer's claims - he is implicitly applying one model, and then acting as though no assumption was made.

"There are various other games you can also play with certainty effects. For example, if you offer someone a certainty of $400, or an 80% probability of $500 and a 20% probability of $300, they'll usually take the $400. But if you ask people to imagine themselves $500 richer, and ask if they would prefer a certain loss of $100 or a 20% chance of losing $200, they'll usually take the chance of losing $200. Same probability distribution over outcomes, different descriptions, different choices."

Ok lets represent this more clearly. a1 - 100% chance to win $400 a2 - 80% chance to win $500 and 20% chance to win $300

b1 - 100% chance to win $500 and 100% chance to lose $100 b2 - 100% chance to win $500 and 20% chance to lose 200%

Lets write it out using utility functions.

a1 - 100%U[$400] a2 - 80%U[$500] + 20%*U[$300]

b1 - 100%U[$500] + 100%U[-$100]? b2 - 100%U[$500] + 20%U[-200%}?

Wait a minute. The probabilities don't add up to one. Maybe I haven't phrased the description correctly. Lets try that again.

b1 - 100% chance to both win $500 and lose $100 b2 - 20% chance both win $500 and to lose $200, leaving an 80% chance to win $500 and lose $0

b1 - 100%U[$500 - $100] = 100%U[$400] b2 - 20%U[$500-$200] + 80%[$500-$0] = 80%U[$500] + 20%U[$300]

This is exactly the same thing as a1 and a2. More importantly however is that the $500 is just a value used to calculate what to plug into the utility function. The $500 by itself has no probability coefficient and therefore it's 'certainty' is irrelevant to the problem at hand. It's a trick using clever wordplay to make one believe there is a 'certainty' when none is there. It's not the same as the Allais paradox.

As for the Allais paradox, I'll have to take another look at it later today.

Eliezer, You need to specify if it's a one-time choice or if it will be repeated. You need to specify if lives or dollars are at stake. These things matter.

Maybe I'm missing something obvious, but doesn't diminishing marginal utility play a big role here? After all, almost all of us would prefer $1,000,000 with certainty to $2,000,100 with 50% probability, and it would be perfectly rational to do so -- not because of the "utility of certainty," but because $2 million isn't quite twice as good as $1 million (for most people). But if you offered us this same choice a thousand times, we would probably then take the $20,000,100, because the many coin flips would reduce the variance enough to create a higher expected utility, even with diminishing marginal returns. (If the math doesn't quite seem to work out, you could probably work out numbers that would.)

So it seems at least plausible that you could construct versions of the money pump problem where you could rationally prefer bid A to bid B in a one-off shot, but where you would then change your preference to bid B if offered multiple times. Obviously I'm not saying that's what's really going on -- Allais paradox surely does demonstrate a real and problematic inconsistency. But we shouldn't conclude from that it's always rational to just "shut up and multiply," at least when we're talking about anything other than "raw" utility.

"I can cause you to invert your preferences over time and pump some money out of you."

I think the small qualifier you slipped in there, "over time", is more salient than it appears at first.

Like most casually intuitive humans, I'll prefer 1A over 1B, and (for the sake of this argument) 2B over 2A, and you can pump some money out of me for a bit.

But... as a somewhat rational thinker, you won't be able to pump an unbounded amount of money out of me. Eventually I catch on to what you're doing and your trickle of cents will disappear. I will go, "well, I don't know what's wrong with my feeble intuition, but I can tell that Elizer is going to end up with all my money this way, so I'll stop even though it goes against my intuition." If you want to accelerate this, make the stuff worth more than a cent. Tell someone that the "mathematically wrong choice will cost you $1,000,000", and I bet they'll take some time to think and choose a set of beliefs that can't be money-pumped.

Or, change the time aspect. I suspect if I were immortal (or at least believed myself to be), I would happily choose 1B over 1A, and certainty be screwed. Maybe I don't get the money, so what, I have an infinite amount of time to earn it back. It's the fact that I don't get to play the game an unlimited amount of times that makes certainty a more valuable aspect.

This appears to be (to my limited knowledge of what science knows a well-known bias. But like most biases, I think I can imagine occasions when it serves as a heuristic.

The thought occurred to me because I play miniature and card games--I see other commenters have also mentioned some games.

Let's say, for example, I have a pair of cards that both give me X of something--let's it deals a certain amount of damage, for those familiar with these games. One card gives me 4 of that something. The other gives me 1-8 over a uniform random distribution--maybe a die roll.

Experience players of these games will tell you that unless the random card gives you a higher expected value, you should play the certain card. And empirical evidence would seem to suggest that they know what they're talking about, because these are the players who win games. What do they say if you ask them why? They say you can plan around the certain gain.

I think that notion is important here. If I have a gain that is certain, at least in any of these games, I can exploit it to its fullest potential--for a high final utility. I can lure my opponent into a trap because I know I can beat them, I can make an aggressive move that only works if I deal at least four damage--heck, the mere ability to trim down my informal Minimax tree is no small gain in a situation like this.

Dealing 4 damage without exploiting it has a much smaller end payoff. And sure, I could try to exploit the random effect in just the same way--I'll get the same effect if I win my roll. But if I TRY to exploit that gain and FAIL, I'll be punished severely. If you add in these values it skews the decision matrix quite a bit.

And none of this is to say that the gambling outcomes being used as examples above aren't what they seem to be. But I'm wondering if humans are bad at these decisions partly because the ancestral environment contained many examples of situations like the one I've described. Trying to exploit a hunting technique that MIGHT work could get you eaten by a bear--a high negative utility hidden in that matrix. And this could lead, after natural selection, to humans who account for such 'hidden' downsides even when they don't exist.

I should have read this post before replying on the last I suppose! Things are a little more clear.

Hmm... well I had more written but for brevity's sake: I suppose my preference system looks more like 1A>1B, 2A=2B. I don't really have a strong preference for an extra 1% vs an extra $3k either way.

The pump really only functions when it is repeated plays; however in that case I'd take 1B instead of 1A.

Potentially a way in which this heuristic makes sense in real world is if the utility of 0$ was negative. If I were an banana seller then if I sold nothing, when I got my next shipment of bananas I wouldn't have enough space in my warehouse and I'd have to throw out some bananas. In this case, I have to take out an insurance policy against 0$ in all cases except certainty. This will hold even if insurance costs are proportional to probability of 0$ if there were fixed transaction costs to buying insurance. 

While this is a mathematically correct argument, it implies a real world relationship that does not hold in all cases. To illustrate while a platonic agent would be correct to select 1b let's instead imagine an impoverished Somalian living on a few dollars a day. 24,000 dollars would be a life-changing amount of money. The increased utility of $3000 additional dollars would in no way make up for a 1/34 chance of losing such an opportunity.

Conversely imagine a billionaire in the same situation. For him the optimal choice is neither 1a or 1b instead he should turn around and leave instead of wasting his time on such a small sum of money. For a middle class American 1b would indeed be the correct choice. So you see naive rationality can be very irrational.

For another example consider the lottery. Occasionally the pot increases such that you're expected return is more than one dollar per dollar spent. In such a case you should still not buy lottery tickets, because your chance of winning back your money is still tiny. The increased size of the pot does not provide much more utility than its normal size for most lottery players. Of course if you possess a lot of capital and there are no rules against it you might make a return with a large investment but such possibilities are normally blocked.

My examples are purposefully extreme, but they work to illustrate that you can't just translate mathematical probabilities directly into real life and call that rational. Real conditions have to be taken into account.

I would always choose the highest expected value if the bet can be repeated and I can withstand the losses. But I can rationalize the preference for certainty in a couple of ways and these might be the same psychology. A: once you said 100%. I treat it as my $24000 and loss aversion might prevent me from betting it to win $3000 despite the favorable odds. B: I don't want to live with the regret of not winning $24000 even if it only happens 3% of the time, whereas I can't really tell the difference 1% makes thus I can't attribute any regret to my choice between 2a and 2b.