Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

The Joy of Bias

14 estimator 09 June 2015 07:04PM

What do you feel when you discover that your reasoning is flawed? when you find your recurring mistakes? when you find that you have been doing something wrong for quite a long time?

Many people feel bad. For example, here is a quote from a recent article on LessWrong:

By depicting the self as always flawed, and portraying the aspiring rationalist's job as seeking to find the flaws, the virtue of perfectionism is framed negatively, and is bound to result in negative reinforcement. Finding a flaw feels bad, and in many people that creates ugh fields around actually doing that search, as reported by participants at the Meetup.

But actually, when you find a serious flaw of yours, you should usually jump for joy. Here's why.

continue reading »

Confound it! Correlation is (usually) not causation! But why not?

44 gwern 09 July 2014 03:04AM

It is widely understood that statistical correlation between two variables ≠ causation. But despite this admonition, people are routinely overconfident in claiming correlations to support particular causal interpretations and are surprised by the results of randomized experiments, suggesting that they are biased & systematically underestimating the prevalence of confounds/common-causation. I speculate that in realistic causal networks or DAGs, the number of possible correlations grows faster than the number of possible causal relationships. So confounds really are that common, and since people do not think in DAGs, the imbalance also explains overconfidence.

Full article: http://www.gwern.net/Causality

Biased Pandemic

56 freyley 13 March 2012 11:32PM

Recently, Portland Lesswrong played a game that was a perfect trifecta of: difficult mental exercise; fun; and an opportunity to learn about biases and recognize them in yourself and others. We're still perfecting it, and we'd welcome feedback, especially from people who try it.

The Short Version

The game is a combination of Pandemic, a cooperative board game that is cognitively demanding, and the idea of roleplaying cognitive biases. Our favorite way of playing it (so far), everyone selects a bias at random, and then attempts to exaggerate that bias in their arguments and decisions during the game. Everyone attempts to identify the biases in the other players, and, when a bias is guessed, the guessed player selects a new bias and begins again.

continue reading »

The Substitution Principle

69 Kaj_Sotala 28 January 2012 04:20AM

Partial re-interpretation of: The Curse of Identity
Also related to: Humans Are Not Automatically Strategic, The Affect Heuristic, The Planning Fallacy, The Availability Heuristic, The Conjunction Fallacy, Urges vs. Goals, Your Inner Google, signaling, etc...

What are the best careers for making a lot of money?

Maybe you've thought about this question a lot, and have researched it enough to have a well-formed opinion. But the chances are that even if you hadn't, some sort of an answer popped into your mind right away. Doctors make a lot of money, maybe, or lawyers, or bankers. Rock stars, perhaps.

You probably realize that this is a difficult question. For one, there's the question of who we're talking about. One person's strengths and weaknesses might make them more suited for a particular career path, while for another person, another career is better. Second, the question is not clearly defined. Is a career with a small chance of making it rich and a large chance of remaining poor a better option than a career with a large chance of becoming wealthy but no chance of becoming rich? Third, whoever is asking this question probably does so because they are thinking about what to do with their lives. So you probably don't want to answer on the basis of what career lets you make a lot of money today, but on the basis of which one will do so in the near future. That requires tricky technological and social forecasting, which is quite difficult. And so on.

Yet, despite all of these uncertainties, some sort of an answer probably came to your mind as soon as you heard the question. And if you hadn't considered the question before, your answer probably didn't take any of the above complications into account. It's as if your brain, while generating an answer, never even considered them.

The thing is, it probably didn't.

Daniel Kahneman, in Thinking, Fast and Slow, extensively discusses what I call the Substitution Principle:

If a satisfactory answer to a hard question is not found quickly, System 1 will find a related question that is easier and will answer it. (Kahneman, p. 97)

System 1, if you recall, is the quick, dirty and parallel part of our brains that renders instant judgements, without thinking about them in too much detail. In this case, the actual question that was asked was ”what are the best careers for making a lot of money”. The question that was actually answered was ”what careers have I come to associate with wealth”.

Here are some other examples of substitution that Kahneman gives:

continue reading »

Amanda Knox: post mortem

23 gwern 20 October 2011 04:10PM

Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:

Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:

  1. the failure of any damning or especially relevant evidence to surface in the ~2 years since (see also: the hope function)
  2. the independent experts’ report on the DNA evidence
  3. the freeing of Knox & Sollecito, and continued imprisonment of Rudy Guede (with reduced sentence)

Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.

continue reading »

The Cognitive Science of Rationality

88 lukeprog 12 September 2011 08:48PM

(The post is written for beginners. Send the link to your friends! Regular Less Wrong readers may want to jump to the Stanovich material.)

The last 40 years of cognitive science have taught us a great deal about how our brains produce errors in thinking and decision making, and about how we can overcome those errors. These methods can help us form more accurate beliefs and make better decisions.


Long before the first Concorde supersonic jet was completed, the British and French governments developing it realized it would lose money. But they continued to develop the jet when they should have cut their losses, because they felt they had "invested too much to quit"1 (sunk cost fallacy2).

John tested positive for an extremely rare but fatal disease, using a test that is accurate 80% of the time. John didn't have health insurance, and the only available treatment — which his doctor recommended — was very expensive. John agreed to the treatment, his retirement fund was drained to nothing, and during the treatment it was discovered that John did not have the rare disease after all. Later, a statistician explained to John that because the disease is so rare, the chance that he had had the disease even given the positive test was less than one in a million. But neither John's brain nor his doctor's brain had computed this correctly (base rate neglect).

Mary gave money to a charity to save lives in the developing world. Unfortunately, she gave to a charity that saves lives at a cost of $100,000 per life instead of one that saves lives at 1/10th that cost, because the less efficient charity used a vivid picture of a starving child on its advertising, and our brains respond more to single, identifiable victims than to large numbers of victims (identifiability effect3 and scope insensitivity4).

During the last four decades, cognitive scientists have discovered a long list of common thinking errors like these. These errors lead us to false beliefs and poor decisions.

How are these errors produced, and how can we overcome them? Vague advice like "be skeptical" and "think critically" may not help much. Luckily, cognitive scientists know a great deal about the mathematics of correct thinking, how thinking errors are produced, and how we can overcome these errors in order to live more fulfilling lives.

continue reading »

Verifying Rationality via RationalPoker.com

32 Louie 25 March 2011 04:32PM

Related to: Problem of verifying rationality

We're excited to announce the (soft) launch of RationalPoker.com! It's a new guide developed by me, Zvi, Kevin, and patrissimo detailing how to use online poker as rationality training to conquer your cognitive biases. We want our community to go from knowing a lot about cognitive biases to actually having a training method that allows us to integrate that knowledge into our habits -- truly reducing biases instead of just leaving us perpetually lamenting our flawed brain-ware. In the coming weeks, we'll be making the case that online poker is a useful rationalist pursuit along with developing introductory "How To" material that allows those who join us to play profitably.

We want to make sure we aren’t wasting our time practicing an ungrounded art with methods that don’t work. Poker gives us an objective way to test x-rationality. The difference between winning and losing in poker once you know a small amount of domain-specific knowledge is due to differing levels of rationality. Our site will be presenting the case that a strong rationalist who can act on their knowledge of cognitive biases (a defining feature of x-rationality but not traditional rationality) should have a distinct advantage. We'll be offering the connecting material between the sequences and online poker to teach you how to apply knowledge of cognitive biases to poker in a way that verifies your current level of rationality and naturally teaches you to improve your rationality over time.

Incidentally, this also presents a solution for those of us looking to earn money from anywhere with a flexible schedule that leaves time for outside interests.

continue reading »

Fun and Games with Cognitive Biases

62 Cosmos 18 February 2011 08:38PM

You may have heard about IARPA's Sirius Program, which is a proposal to develop serious games that would teach intelligence analysts to recognize and correct their cognitive biases.  The intelligence community has a long history of interest in debiasing, and even produced a rationality handbook based on internal CIA publications from the 70's and 80's.  Creating games which would systematically improve our thinking skills has enormous potential, and I would highly encourage the LW community to consider this as a potential way forward to encourage rationality more broadly.

While developing these particular games will require thought and programming, the proposal did inspire the NYC LW community to play a game of our own.  Using a list of cognitive biases, we broke up into groups of no larger than four, and spent five minutes discussing each bias with regards to three questions:

  1. How do we recognize it?
  2. How do we correct it?
  3. How do we use its existence to help us win?

The Sirius Program specifically targets Confirmation Bias, Fundamental Attribution Error, Bias Blind Spot, Anchoring Bias, Representativeness Bias, and Projection Bias.  To this list, I also decided to add the Planning Fallacy, the Availability Heuristic, Hindsight Bias, the Halo Effect, Confabulation, and the Overconfidence Effect.  We did this Pomodoro style, with six rounds of five minutes, a quick break, another six rounds, before a break and then a group discussion of the exercise.

Results of this exercise are posted below the fold.  I encourage you to try the exercise for yourself before looking at our answers.

continue reading »

Human performance, psychometry, and baseball statistics

24 Craig_Heldreth 15 October 2010 01:13PM

I. Performance levels and age

Human ambition for achievement in modest measure gives meaning to our lives, unless one is an existentialist pessimist like Schopenhauer who taught that life with all its suffering and cruelty simply should not be. Psychologists study our achievements under a number of different descriptions--testing for IQ, motivation, creativity, others. As part of my current career transition, I have been examining my own goals closely, and have recently read a fair amount on these topics which are variable in their evidence.

A useful collection of numerical data on the subject of human performance is the collection of Major League Baseball player performance statistics--the batting averages, number home runs, runs batted in, slugging percentage--of the many thousands of participants in the hundred years since detailed statistical records have been kept and studied by the players, journalists, and fans of the sport. The advantage of examining issues like these from the angle of Major League Baseball player performance statistics is the enormous sample size of accurately measured and archived data.

The current senior authority in this field is Bill James, who now works for the Boston Red Sox; for the first twenty-five years of his activity as a baseball statistician James was not employed by any of the teams. It took him a long time to find a hearing for his views on the inside of the industry, although the fans started buying his books as soon as he began writing them.

In one of the early editions of his Baseball Abstract, James discussed the biggest fallacies that managers and executives held regarding the achievements of baseball players. He was adamant about the most obvious misunderstood fact of player performance: it is sharply peaked at age 27 and decreases rapidly, so rapidly that only the very best players were still useful at the age of 35. He was able to observe only one executive that seemed to intuit this--a man whose sole management strategy was to trade everybody over the age of 30 for the best available player under the age of 30 he could acquire.

continue reading »

What Intelligence Tests Miss: The psychology of rational thought

36 Kaj_Sotala 11 July 2010 11:01PM

This is the fourth and final part in a mini-sequence presenting Keith E. Stanovich's excellent book What Intelligence Tests Miss: The psychology of rational thought.

If you want to give people a single book to introduce people to the themes and ideas discussed on Less Wrong, What Intelligence Tests Miss is probably the best currenty existing book for doing so. It does have a somewhat different view on the study of bias than we on LW: while Eliezer concentrated on the idea of the map and the territory and aspiring to the ideal of a perfect decision-maker, Stanovich's perspective is more akin to bias as a thing that prevents people from taking full advantage of their intelligence. Regardless, for someone less easily persuaded by LW's somewhat abstract ideals, reading Stanovich's concrete examples first and then proceeding to the Sequences is likely to make the content presented in the sequences much more interesting. Even some of our terminology such as "carving reality at the joints" and the instrumental/epistemic rationality distinction will be more familiar to somebody who was first read What Intelligence Tests Miss.

Below is a chapter-by-chapter summary of the book.

Inside George W. Bush's Mind: Hints at What IQ Tests Miss is a brief introductory chapter. It starts with the example of president George W. Bush, mentioning that the president's opponents frequently argued against his intelligence, and even his supporters implicitly conceded the point by arguing that even though he didn't have "school smarts" he did have "street smarts". Both groups were purportedly surprised when it was revealed that the president's IQ was around 120, roughly the same as his 2004 presidential candidate opponent John Kerry. Stanovich then goes on to say that this should not be surprising, for IQ tests do not tap into the tendency to actually think in an analytical manner, and that IQ had been overvalued as a concept. For instance, university admissions frequently depend on tests such as the SAT, which are pretty much pure IQ tests. The chapter ends by a disclaimer that the book is not an attempt to say that IQ tests measure nothing important, or that there would be many kinds of intelligence. IQ does measure something real and important, but that doesn't change the fact that people overvalue it and are generally confused about what it actually does measure.

Dysrationalia: Separating Rationality and Intelligence talks about the phenomenon informally described as "smart but acting stupid". Stanovich notes that if we used a broad definition of intelligence, where intelligence only meant acting in an optimal manner, then this expression wouldn't make any sense. Rather, it's a sign that people are intuitively aware of IQ and rationality as measuring two separate qualities. Stanovich then brings up the concept of dyslexia, which the DSM IV defines as "reading achievement that falls substantially below that expected given the individual's chronological age, measured intelligence, and age-appropriate education". Similarly, the diagnostic criterion for mathematics disorder (dyscalculia) is "mathematical ability that falls substantially below that expected for the individual's chronological age, measured intelligence, and age-appropriate education". He argues that since we have a precedent for creating new disability categories when someone's ability in an important skill domain is below what would be expected for their intelligence, it would make sense to also have a category for "dysrationalia":

Dysrationalia is the inability to think and behave rationally despite adequate intelligence. It is a general term that refers to a heterogenous group of disorders manifested by significant difficulties in belief formation, in the assessment of belief consistency, and/or in the determination of action to achieve one's goals. Although dysrationalia may occur concomitantly with other handicapping conditions (e.g. sensory impairment), dysrationalia is not the result of those conditions. The key diagnostic criterion for dysrationalia is a level of rationality, as demonstrated in thinking and behavior, that is significantly below the level of the individual's intellectual capacity (as determined by an individually administered IQ test).

continue reading »

View more: Next