Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Reference Frames for Expected Value

3 ozziegooen 16 March 2014 07:22PM

Puzzle 1: George mortgages his house to invest in lottery tickets. He wins and becomes a millionaire. Did he make a good choice?

Puzzle 2: The U.S. president questions if he should bluff a nuclear war or concede to the USSR. He bluffs and it just barely works.  Although there were several close calls for nuclear catastrophe, everything works out ok. Was this ethical?

One interpretation of consequentialism is that decisions that produce good outcomes are good decisions, rather than decisions that produce good expected outcomes.12 One would be ethical if their actions end up with positive outcomes, disregarding the intentions of those actions. For instance, a terrorist who accidentally foils an otherwise catastrophic terrorist plan would have done a very ‘morally good’ action.3 This general view seems to be surprisingly common.4

This seems intuitively strange to many, it definitely is to me. Instead, ‘expected value’ seems to be a better way of both making decisions and judging the decisions made by others. However, while ‘expected value’ can be useful for individual decision making, I make the case that it is very difficult to use to judge other people’s decisions in a meaningful way.5 This is because ‘expected value’ is typically defined in reference to a specific set of information and intelligence rather than an objective truth about the world.

Two questions to help guide this:

  1. Should we judge previous actions based on ‘expected’ or ‘actual’ value?
  2. Should we make future decisions to optimize ‘expected’ or ‘actual’ value?

I believe these are in a sense quite simple, but require some consideration to definitions.6

Optimizing Future Decisions: Actual vs. Expected Value

The second question is the easiest of the two, so I’ll begin with that one. The simple answer is that this is a question of defining ‘expected value’. Once we do so the question kind of goes away.

There is nothing fundamentally different between expected value and actual value.  A more fair comparison may be ‘expected value from the perspective of the decision maker’ with ‘expected value from a later, more accurate prospective’.

Expected value converges on actual value with lots of information. Said differently, actual value is expected value with complete information.

In the case of an individual purchasing lottery tickets successfully (Puzzle 1), the ‘actual value’ is still not exact from our point of view. While we may know how much money was won, or what profit was made. We also don’t know what the counterfactual would have been. It is still theoretically possible that in the worlds where George wouldn’t have purchased the lottery tickets, he would have been substantially better off. While the fact that we have imperfect information doesn’t matter too much, I think it demonstrates that presenting a description of the outcome as ‘actual value’ is incomplete. ‘Actual value’ exists only theoretically, even after the fact.7

So this question becomes, then ‘should one make a decision to optimize value using the information and knowledge available to them, or using perfect knowledge and information?’ Obviously, in this case, ‘perfect knowledge’ is inaccessible to them (or the ‘expected value’ and ‘actual value’ would be the same exact thing). I believe it should be quite apparent that in this case, the best one can do (and should do) is make the best decision using their available information.

This question is similar to asking ‘should you drive your car as quickly as your car can drive, or much faster than your car can drive?’ Obviously you may like to drive faster, but that’s by definition not an option. Another question: ‘should you do well in life or should you become an all-powerful dragon king?’

Judging Previous Decisions: Actual vs. Expected Value

Judging previous decisions can get tricky.

Let’s study the lottery example again. A person purchases a lottery ticket and wins. For simplicity, let’s say the decision to purchase the ticket was done only to optimize money.

The question is, what is the expected value of purchasing the lottery ticket? How does this change depending on information and knowledge?

In general purchasing a lottery ticket can be expected to be a net loss in earnings, and thus a bad decision. However, if one was sure they would win, it would be a pretty good idea. Given the knowledge that the player won, the player made a good decision. Winning the lottery clearly is better than not playing once.

More interesting is considering the limitation not in information about the outcome but about knowledge of probability. Say the player thought that they were likely win the lottery, that it was a good purchase. This may seem insane to someone familiar with probability and the lottery system, but not everyone is familiar with these things.

From the point of view of the player, the lottery ticket purchase had net-positive utility. From the point of view of a person with knowledge of the lottery and/or statistics, the purchase had net-negative utility. From the point of view of any of these two groups, after they know that the lottery will be a success, it was a net positive decision.

  No Knowledge of Outcome Knowledge of Outcome
‘Intelligent’ Person with Knowledge of Probability Negative Positive
Lottery Player Positive Positive

Expected Value of purchasing a Lottery Ticket from different Reference Points

To make things a bit more interesting, imagine that there’s a genius out there with a computer simulation of our exact universe. This person can tell which lottery ticket will win in advance because they can run the simulations. To this ‘genius’ it’s obvious that the purchase is a net-positive outcome.

  No Knowledge of Outcome Knowledge of Outcome
Genius Positive Positive
‘Intelligent’ Person with Knowledge of Probability Negative Positive
Lottery Player Positive Positive

Expected Value of purchasing a Lottery Ticket from different Reference Points

So what is the expected value of purchasing the lottery ticket? The answer is that the ‘expected value’ is completely dependent on the ‘reference frame’, or a specific set of information and intelligence. From the reference frame of the ‘intelligent person’ this was low in expected value, so was a bad decision. From that of the genius, it was a good decision. And from the player, a good decision.


So how do we judge this poor (well, soon rich) lottery player? They made a good decision respective to the results, respective to the genius, and compared to their own knowledge. Should we say ‘oh, this person should have had slightly more knowledge, but not too much knowledge, and thus they made a bad choice’? What does that even mean?

Perhaps we could judge the player for not reading into lottery facts before playing. Wasn’t it irresponsible for falling for such a simple fallacy? Or perhaps the person was ‘lazy’ to not learn probability in the first place.

Well, things like these seem like intuitions to me. We may have the intuitions to us that the lottery is a poor choice. We may find facts to prove these intuitions accurate. But the gambler my not have these intuitions. It seems unfair to consider any intuitions ‘obvious’ to those who do not share them.

One might also say that the gambler probably knew it was a bad idea, but let his or her ‘inner irrationalities’ control the decision process. Perhaps they were trying to take an ‘easy way out’ of some sort. However, these seem quite judgmental as well. If a person experiences strong emotional responses; fear, anger, laziness; those inner struggles would change their expected value calculation. It might be a really bad, heuristically-driven ‘calculation’, but it would be the best they would have at that time.

Free Will Bounded Expected Value

We are getting to the question of free will and determinism. After all, if there is any sort of free will, perhaps we have the ability to make decisions that are sub-optimal by our expected value functions. Perhaps we commonly do so (else it wouldn’t be much in the sense of ‘free’ will.)

This would be interesting because it would imply an ‘expected result’ that the person should have calculated, even if they didn’t actually do so. We need to understand the person’s actions and understanding, not in terms of what we know, or what they knew, but what they should have figured out given their knowledge.

This would require a very well specified Free Will Boundary of some sort. A line around a few thought processes, parts of the brain, and resource constraints, which could produce a thereby optimal expected result calculation. Anything less than this ‘optimal given Free Will Boundary’ expected value calculation would be fair game for judging.

Conclusion: Should we Even Judge People or Decisions Anyway?

So, deciding to make future decisions based on expected value seems reasonable.  The main question in this essay, the harder question, is if we can judge previous decisions based on their respective expected values, and how to possibly come up with the relevant expected values to do so.

I think that we naturally judge people. We have old and modern heroes and villains. Judging people is simply something that humans do. However, I believe that on close inspection this is very challenging if not impossible to do reasonably and precisely.

Perhaps we should attempt to stop placing so much emphasis on individualism and just try to do the best we can while not judging others nor other decisions much. Considerations of judging may be interesting, but the main take away may be the complexity itself, indicated that judgements are very subjective and incredibly messy.

That said, it can still be useful to analyze previous decisions or individuals. That seems like one of the best ways to update our priors of the world. We just need to remember not to treat it personally.

  1. Dorsey, Dale. “Consequentialism, Metaphysical Realism, and the Argument from Cluelessness.” University of Kansas Department of Philosophy http://people.ku.edu/~ddorsey/cluelessness.pdf

  2. Sinhababu, Neiladri. “Moral Luck.” Tedx Presentation http://www.youtube.com/watch?v=RQ7j7TD8PWc

  3. This is assuming the terrorists are trying to produce ‘disutility’ or a value separate from ‘utility’. I feel like from their perspective, maximizing an intrinsic value dissimilar from our notion of utility would be maximizing ‘expected value’. But analyzing the morality of people with alternative value systems is a very different matter.

  4. These people tend not to like consequentialism much.

  5. I don’t want to impose what I deem to be a false individualistic appeal, so consider this to mean that one would have a difficult time judging anyone at any time except for their spontaneous consciousness.

  6. I bring them up because they are what I considered and have talked to others about before understanding what makes them frustrating to answer. Basically, they are nice starting points for getting towards answering the questions that were meant to be asked instead.

  7. This is true for essentially all physical activities. Thought experiments or very simple simulations may be exempt.

SotW: Check Consequentialism

38 Eliezer_Yudkowsky 29 March 2012 01:35AM

(The Exercise Prize series of posts is the Center for Applied Rationality asking for help inventing exercises that can teach cognitive skills.  The difficulty is coming up with exercises interesting enough, with a high enough hedonic return, that people actually do them and remember them; this often involves standing up and performing actions, or interacting with other people, not just working alone with an exercise booklet and a pencil.  We offer prizes of $50 for any suggestion we decide to test, and $500 for any suggestion we decide to adopt.  This prize also extends to LW meetup activities and good ideas for verifying that a skill has been acquired.  See here for details.)

Exercise Prize:  Check Consequentialism

In philosophy, "consequentialism" is the belief that doing the right thing makes the world a better place, i.e., that actions should be chosen on the basis of their probable outcomes.  It seems like the mental habit of checking consequentialism, asking "What positive future events does this action cause?", would catch numerous cognitive fallacies.

For example, the mental habit of consequentialism would counter the sunk cost fallacy - if a PhD wouldn't really lead to much in the way of desirable job opportunities or a higher income, and the only reason you're still pursuing your PhD is that otherwise all your previous years of work will have been wasted, you will find yourself encountering a blank screen at the point where you try to imagine a positive future outcome of spending another two years working toward your PhD - you will not be able to state what good future events happen as a result.

Or consider the problem of living in the should-universe; if you're thinking, I'm not going to talk to my boyfriend about X because he should know it already, you might be able to spot this as an instance of should-universe thinking (planning/choosing/acting/feeling as though within / by-comparison-to an image of an ideal perfect universe) by having done exercises specifically to sensitize you to should-ness.  Or, if you've practiced the more general skill of Checking Consequentialism, you might notice a problem on asking "What happens if I talk / don't talk to my boyfriend?" - providing that you're sufficiently adept to constrain your consequentialist visualization to what actually happens as opposed to what should happen.


The skill of Checking Consequentialism isn't quite as simple as telling people to ask, "What positive result do I get?"  By itself, this mental query is probably going to return any apparent justification - for example, in the sunk-cost-PhD example, asking "What good thing happens as a result?" will just return, "All my years of work won't have been wasted!  That's good!"  Any choice people are tempted by seems good for some reason, and executing a query about "good reasons" will just return this.

The novel part of Checking Consequentialism is the ability to discriminate "consequentialist reasons" from "non-consequentialist reasons" - being able to distinguish that "Because a PhD gets me a 50% higher salary" talks about future positive consequences, while "Because I don't want my years of work to have been wasted" doesn't.

continue reading »

Consequentialism Need Not Be Nearsighted

53 orthonormal 02 September 2011 07:37AM

Summary: If you object to consequentialist ethical theories because you think they endorse horrible or catastrophic decisions, then you may instead be objecting to short-sighted utility functions or poor decision theories.

Recommended: Decision Theory Paradox: PD with Three Implies Chaos?

Related: The "Intuitions" Behind "Utilitarianism", Yvain's Consequentialism FAQ

The simple idea that we ought to choose actions according to their probable consequences, ever since it was formulated, has garnered a rather shocking amount of dissent. Part of this may be due to causes other than philosophical objections, and some of the objections get into the metaphysics of metaethics. But there's a fair amount of opposition on rather simple grounds: that consequentialist reasoning appears to endorse bad decisions, either in the long run or as an effect of collective action.

Every so often, you'll hear someone offer a reductio ad absurdum of the following form: "Consider dilemma X. If we were consequentialists, then we would be forced to choose Y. But in the long run (or if widely adopted) the strategy of choosing Y leads to horrible consequence Z, and so consequentialism fails on its own terms."

There's something fishy about the argument when you lay it out like that: if it can be known that the strategy of choosing Y has horrible consequence Z, then why do we agree that consequentialists choose Y? In fact, there are two further unstated assumptions in every such argument I've heard, and it is those assumptions rather than consequentialism on which the absurdity really falls. But to discuss the assumptions, we need to delve into a bit of decision theory.

continue reading »

Ethics has Evidence Too

21 Jack 06 February 2010 06:28AM

A tenet of traditional rationality is that you can't learn much about the world from armchair theorizing. Theory must be epiphenomenal to observation-- our theories are functions that tell us what experiences we should anticipate, but we generate the theories from *past* experiences. And of course we update our theories on the basis of new experiences. Our theories respond to our evidence, usually not the other way around. We do it this way because it works better then trying to make predictions on the basis of concepts or abstract reasoning. Philosophy from Plato through Descartes and to Kant is replete with failed examples of theorizing about the natural world on the basis of something other than empirical observation. Socrates thinks he has deduced that souls are immortal, Descartes thinks he has deduced that he is an immaterial mind, that he is immortal, that God exists and that he can have secure knowledge of the external world, Kant thinks he has proven by pure reason the necessity of Newton's laws of motion.

These mistakes aren't just found in philosophy curricula. There is a long list of people who thought they could deduce Euclid's theorems as analytic or a priori knowledge. Epicycles were a response to new evidence but they weren't a response that truly privileged the evidence. Geocentric astronomers changed their theory *just enough* so that it would yield the right predictions instead of letting a new theory flow from the evidence. Same goes for pre-Einsteinian theories of light. Same goes for quantum mechanics. A kludge is a sign someone is privileging the hypothesis. It's the same way many of us think the Italian police changed their hypothesis explaining the murder of Meredith Kercher once it became clear Lumumba had an alibi and Rudy Guede's DNA and hand prints were found all over the crime scene. They just replaced Lumumba with Guede and left the rest of their theory unchanged even though there was no longer reason to include Knox and Sollecito in the explanation of the murder. These theories may make it over the bar of traditional rationality but they sail right under what Bayes theorem requires.

Most people here get this already and many probably understand it better than I do. But I think it needs to be brought up in the context of our ongoing discussion of normative ethics.

Unless we have reason to think about ethics differently, our normative theories should respond to evidence in the same way we expect our theories in other domains to respond to evidence. What are the experiences that we are trying to explain with our ethical theories? Why bother with ethics at all? What is the mystery we are trying to solve? The only answer I can think of is our ethical intuitions. When faced with certain situations in real life or in fiction we get strong impulses to react in certain ways, to praise some parties and condemn others. We feel guilt and sometimes pay amends. There are some actions which we have a visceral abhorrence of.

These reactions are for ethics what measurements of time and distance are for physics -- the evidence.

continue reading »

Deontology for Consequentialists

46 Alicorn 30 January 2010 05:58PM

Consequentialists see morality through consequence-colored lenses.  I attempt to prise apart the two concepts to help consequentialists understand what deontologists are talking about.

Consequentialism1 is built around a group of variations on the following basic assumption:

  • The rightness of something depends on what happens subsequently.

It's a very diverse family of theories; see the Stanford Encyclopedia of Philosophy article.  "Classic utilitarianism" could go by the longer, more descriptive name "actual direct maximizing aggregative total universal equal-consideration agent-neutral hedonic act2 consequentialism".  I could even mention less frequently contested features, like the fact that this type of consequentialism doesn't have a temporal priority feature or side constraints.  All of this is is a very complicated bag of tricks for a theory whose proponents sometimes claim to like it because it's sleek and pretty and "simple".  But the bottom line is, to get a consequentialist theory, something that happens after the act you judge is the basis of your judgment.

To understand deontology as anything but a twisted, inexplicable mockery of consequentialism, you must discard this assumption.

Deontology relies on things that do not happen after the act judged to judge the act.  This leaves facts about times prior to and the time during the act to determine whether the act is right or wrong.  This may include, but is not limited to:

  • The agent's epistemic state, either actual or ideal (e.g. thinking that some act would have a certain result, or being in a position such that it would be reasonable to think that the act would have that result)
  • The reference class of the act (e.g. it being an act of murder, theft, lying, etc.)
  • Historical facts (e.g. having made a promise, sworn a vow)
  • Counterfactuals (e.g. what would happen if others performed similar acts more frequently than they actually do)
  • Features of the people affected by the act (e.g. moral rights, preferences, relationship to the agent)
  • The agent's intentions (e.g. meaning well or maliciously, or acting deliberately or accidentally)
continue reading »

The Lifespan Dilemma

39 Eliezer_Yudkowsky 10 September 2009 06:45PM

One of our most controversial posts ever was "Torture vs. Dust Specks".  Though I can't seem to find the reference, one of the more interesting uses of this dilemma was by a professor whose student said "I'm a utilitarian consequentialist", and the professor said "No you're not" and told them about SPECKS vs. TORTURE, and then the student - to the professor's surprise - chose TORTURE.  (Yay student!)

In the spirit of always making these things worse, let me offer a dilemma that might have been more likely to unconvince the student - at least, as a consequentialist, I find the inevitable conclusion much harder to swallow.

continue reading »

Protein Reinforcement and DNA Consequentialism

27 Eliezer_Yudkowsky 13 November 2007 01:34AM

Followup toEvolutionary Psychology

It takes hundreds of generations for a simple beneficial mutation to promote itself to universality in a gene pool.  Thousands of generations, or even millions, to create complex interdependent machinery.

That's some slow learning there.  Let's say you're building a squirrel, and you want the squirrel to know locations for finding nuts.  Individual nut trees don't last for the thousands of years required for natural selection.  You're going to have to learn using proteins.  You're going to have to build a brain.

continue reading »