Decision Analysis Sequence

24 Vaniver 21 December 2011 12:12AM

This is the introduction (conclusion) to my decision analysis sequence. It covers (much more quickly and less completely) what you would expect to see in a semester-long course on decision making. The posts are:

  1. Uncertainty: the basics of treating uncertainties as probabilities and doing Bayesian math.
  2. 5 Axioms of Decision Making: the five steps / assumptions that form the foundation of careful decision-making.
  3. Compressing Reality to Math: how to take a sticky, complicated situation and condense it down to something a calculator can solve, without feeling like you've left something important out.
  4. Measures, Risk, Death, and War: how to deal with many similar prospects (utilities), risks of death, and adversaries.
  5. Value of Information: Four Examples: how to value information-gathering activity, like tests or waiting, and incorporate it into your decision-making process.

I'd like to welcome any comments about the sequence here. What parts did I do well? What parts need work? What parts would you like to see expanded (or removed)?

One of the difficulties in posting about a topic like this is that it's foundational: basic, but important to get right. The idea of an expected utility calculation is not new (although the approach I take here may be novel for many of you) and, like I say in the VoI post, there's often more benefit in applying the process to examples than repeatedly talking about the process. The case studies I have access to, though, are not ones I can publish online, and I don't think I can construct an example that would work as well as a real one. Do people have problems they would like me to analyze with this framework as examples?

Measures, Risk, Death, and War

11 Vaniver 20 December 2011 11:37PM

This is the fourth post of a sequence on decision analysis, preceded by Compressing Reality to Math. It touches on a wide variety of topics which didn't seem to work well as posts of their own, either because they were too short or too long.

continue reading »

Compressing Reality to Math

20 Vaniver 15 December 2011 12:07AM

This is part of a sequence on decision analysis and follows 5 Axioms of Decision-Making, which explains how to turn a well-formed problem into a solution. Here we discuss turning reality into a well-formed problem. There are three basic actions I'd like to introduce, and then work through some examples.

continue reading »

Funnel plots: the study that didn't bark, or, visualizing regression to the null

47 gwern 04 December 2011 11:05AM

Marginal Revolution linked a post at Genomes Unzipped, "Size matters, and other lessons from medical genetics", with the interesting centerpiece graph:

a funnel plot of genetic studies showing null result approached as sample size increasese

This is from pg 3 of an Ioannidis 2001 et al article (who else?) on what is called a funnel plot: each line represents a series of studies about some particularly hot gene-disease correlations, plotted where Y =  the odds ratio (measure of effect size; all results are 'statistically significant', of course) and X = the sample size. The 1 line is the null hypothesis, here. You will notice something dramatic: as we move along the X-axis and sample sizes increase, everything begins to converge on 1:

Readers familiar with the history of medical association studies will be unsurprised by what happened over the next few years: initial excitement (this same polymorphism was associated with diabetes! And longevity!) was followed by inconclusive replication studies and, ultimately, disappointment. In 2000, 8 years after the initial report, a large study involving over 5,000 cases and controls found absolutely no detectable effect of the ACE polymorphism on heart attack risk. In the meantime, the same polymorphism had turned up in dozens of other association studies for a wide range of traits ranging from obstet­ric cholestasis to menin­go­­coccal disease in children, virtually none of which have ever been convincingly replicated.

(See also "Why epidemiology will not correct itself" or the DNB FAQ.)

continue reading »

5 Axioms of Decision Making

35 Vaniver 01 December 2011 10:22PM

This is part of a sequence on decision analysis; the first post is a primer on Uncertainty.

Decision analysis has two main parts: abstracting a real situation to math, and then cranking through the math to get an answer. We started by talking a bit about how probabilities work, and I'll finish up the inner math in this post. We're working from the inside out because it's easier to understand the shell once you understand the kernel. I'll provide an example of prospects and deals to demonstrate the math, but first we should talk about axioms. In order to be comfortable with using this method, there are five axioms1 you have to agree with, and if you agree with those axioms, then this method flows naturally. They are: Probability, Order, Equivalence, Substitution, and Choice.

continue reading »

Uncertainty

20 Vaniver 29 November 2011 11:12PM

This is part of a sequence on decision analysis.

Decision-making under certainty is pretty boring. You know exactly what each choice will do, and so you order the outcomes based on your preferences, and pick the action that leads to the best outcome.

Human decision-making, though, is made in the presence of uncertainty. Decision analysis - careful decision making - is all about coping with the existence of uncertainty.

Some terminology: a distinction is something uncertain; an event is each of the possible outcomes of that distinction; a prospect is an event that you have a personal stake in, and a deal is a distinction over prospects. This post will focus on distinctions and events. If you're comfortable with probability just jump to the four bolded questions and make sure you get the answers right. Deals are the interesting part, but require this background.

continue reading »

Amanda Knox: post mortem

23 gwern 20 October 2011 04:10PM

Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:

Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:

  1. the failure of any damning or especially relevant evidence to surface in the ~2 years since (see also: the hope function)
  2. the independent experts’ report on the DNA evidence
  3. the freeing of Knox & Sollecito, and continued imprisonment of Rudy Guede (with reduced sentence)

Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.

continue reading »

Rational Home Buying

99 Yvain 27 August 2011 12:15AM

My parents are considering moving house. I've had a front-seat window to their decision process as they compare alternatives, and sometimes it isn't pretty.

A new house is one of the most important purchases most people will make. Because of the sums involved, the usual pitfalls of decision-making gain new importance, and it becomes especially important to make sure you're thinking rationally. Research in a couple of fields, most importantly positive psychology, offers some potentially helpful tips.

continue reading »

The True Rejection Challenge

43 Alicorn 27 June 2011 07:18AM

An exercise:

Name something that you do not do but should/wish you did/are told you ought, or that you do less than is normally recommended.  (For instance, "exercise" or "eat vegetables".)

Make an exhaustive list of your sufficient conditions for avoiding this thing.  (If you suspect that your list may be non-exhaustive, mention that in your comment.)

Precommit that: If someone comes up with a way to do the thing which doesn't have any of your listed problems, you will at least try it.  It counts if you come up with this response yourself upon making your list.

(Based on: Is That Your True Rejection?)

Edit to add: Kindly stick to the spirit of the exercise; if you have no advice in line with the exercise, this is not the place to offer it.  Do not drift into confrontational or abusive demands that people adjust their restrictions to suit your cached suggestion, and do not offer unsolicited other-optimizing.

To alleviate crowding, Armok_GoB has created a second thread for this challenge.

Shifting Load to Explicit Reasoning

13 Vladimir_Nesov 07 May 2011 06:00PM

Related to: Which Parts Are "Me"?, Making your explicit reasoning trustworthy, The 5-Second Level.

What's damaging about moralizing that we wish to avoid, what useful purpose does moralizing usually serve, and what allows to avoid the damage while retaining the usefulness? It engages psychological adaptations that promote conflict (by playing on social status), which are unpleasant to experience and can lead to undesirable consequences in the long run (such as feeling systematically uncomfortable interacting with a person, and so not being able to live or work or be friends with them). It serves the purpose of imprinting your values, which you feel to be right, on the people you interact with. Consequentialist elucidation of reasons for approving or disapproving of a given policy (virtue) is an effective persuasion technique if your values are actually right (for the people you try to confer them on), and it doesn't engage the same parts of your brain that make moralizing undesirable.

What happens here is transfer of responsibility for important tasks from the imperfect machinery that historically used to manage them (with systematic problems in any given context that humans but not evolution can notice), to explicit reasoning.

continue reading »

View more: Prev | Next