This is the fourth and final part in a mini-sequence presenting Keith E. Stanovich's excellent book What Intelligence Tests Miss: The psychology of rational thought.

If you want to give people a single book to introduce people to the themes and ideas discussed on Less Wrong, What Intelligence Tests Miss is probably the best currenty existing book for doing so. It does have a somewhat different view on the study of bias than we on LW: while Eliezer concentrated on the idea of the map and the territory and aspiring to the ideal of a perfect decision-maker, Stanovich's perspective is more akin to bias as a thing that prevents people from taking full advantage of their intelligence. Regardless, for someone less easily persuaded by LW's somewhat abstract ideals, reading Stanovich's concrete examples first and then proceeding to the Sequences is likely to make the content presented in the sequences much more interesting. Even some of our terminology such as "carving reality at the joints" and the instrumental/epistemic rationality distinction will be more familiar to somebody who was first read What Intelligence Tests Miss.

Below is a chapter-by-chapter summary of the book.

Inside George W. Bush's Mind: Hints at What IQ Tests Miss is a brief introductory chapter. It starts with the example of president George W. Bush, mentioning that the president's opponents frequently argued against his intelligence, and even his supporters implicitly conceded the point by arguing that even though he didn't have "school smarts" he did have "street smarts". Both groups were purportedly surprised when it was revealed that the president's IQ was around 120, roughly the same as his 2004 presidential candidate opponent John Kerry. Stanovich then goes on to say that this should not be surprising, for IQ tests do not tap into the tendency to actually think in an analytical manner, and that IQ had been overvalued as a concept. For instance, university admissions frequently depend on tests such as the SAT, which are pretty much pure IQ tests. The chapter ends by a disclaimer that the book is not an attempt to say that IQ tests measure nothing important, or that there would be many kinds of intelligence. IQ does measure something real and important, but that doesn't change the fact that people overvalue it and are generally confused about what it actually does measure.

Dysrationalia: Separating Rationality and Intelligence talks about the phenomenon informally described as "smart but acting stupid". Stanovich notes that if we used a broad definition of intelligence, where intelligence only meant acting in an optimal manner, then this expression wouldn't make any sense. Rather, it's a sign that people are intuitively aware of IQ and rationality as measuring two separate qualities. Stanovich then brings up the concept of dyslexia, which the DSM IV defines as "reading achievement that falls substantially below that expected given the individual's chronological age, measured intelligence, and age-appropriate education". Similarly, the diagnostic criterion for mathematics disorder (dyscalculia) is "mathematical ability that falls substantially below that expected for the individual's chronological age, measured intelligence, and age-appropriate education". He argues that since we have a precedent for creating new disability categories when someone's ability in an important skill domain is below what would be expected for their intelligence, it would make sense to also have a category for "dysrationalia":

Dysrationalia is the inability to think and behave rationally despite adequate intelligence. It is a general term that refers to a heterogenous group of disorders manifested by significant difficulties in belief formation, in the assessment of belief consistency, and/or in the determination of action to achieve one's goals. Although dysrationalia may occur concomitantly with other handicapping conditions (e.g. sensory impairment), dysrationalia is not the result of those conditions. The key diagnostic criterion for dysrationalia is a level of rationality, as demonstrated in thinking and behavior, that is significantly below the level of the individual's intellectual capacity (as determined by an individually administered IQ test).

The Reflective Mind, the Algorithmic Mind, and the Autonomous Mind presents a three-level model of the mind, which I mostly covered in A Taxonomy of Bias: The Cognitive Miser. At the end, we return to the example of George W. Bush, and are shown a bunch of quotes from the president's supporters describing him. His speechwriter called him "sometimes glib, even dogmatic; often uncurious and as a result ill-informed"; John McCain said Bush never asks for his opinion and that the president "wasn't intellectually curious". The same sentiment was echoed by a senior official in Iraq who had observed Bush in various videoconferences and said that the president's "obvious lack of interest in long, detailed discussions, had a chilling effect". On the other hand, other people were quoted as saying that Bush was "extraordinarily intelligent, but was not interested in learning unless it had practical value". Tony Blair repeatedly told his associates that Bush was "very bright". This is taken as evidence that while Bush is indeed intelligent, he does not have thinking dispositions that would have make him make use of his intelligence: he has dysrationalia.

Cutting Intelligence Down to Size further criticizes the trend of treating the word "intelligence" in a manner that is too broad. Stanovich points out that even critics of the IQ concept who introduce terms such as "social intelligence" and "bodily-kinesthetic intelligence" are probably shooting themselves in the foot. By giving everything valuable the label of intelligence, these critics are actually increasing the esteem of IQ tests, and therefore making people think that IQ measures more than it does.

Consider a thought experiment. Imagine that someone objected to the emphasis given to horsepower (engine power) when evaluating automobiles. They feel that horsepower looms too large in people's thinking. In an attempt to deemphasize horsepower, they then being to term the other features of the car things like "braking horsepower" and "cornering horsepower" and "comfort horsepower". Would such a strategy make people less likely to look to engine power as an indicator of the "goodness" of a car? I think not. [...] Just as calling "all good car things" horsepower would emphasize horsepower, I would argue that calling "all good cognitive things" intelligence will contribute to the deification of MAMBIT [Mental Abilities Measured By Intelligence Tests].

Stanovich then continues to argue in favor of separating rationality and intelligence, citing surveys that suggest that folk psychology does already distinguish between the two. He also brings up the chilling effect that deifying intelligence seems to be having on society. Reviews about a book discussing the maltreatment of boys labeled feebleminded seemed to concentrate on the stories of the boys who were later found to have normal IQs, implying that abusive treatment of boys who actually did have a low IQ was okay. Various parents seem to take a diagnosis of low mental ability as much more shocking than a diagnosis such as ADHD or learning disability that stresses the presence of normal IQ, even though the life problems associated with some emotional and behavior disorders are much more severe than those associated with many forms of moderate or mild intellectual disability.

Why Intelligent People Doing Foolish Things Is No Surprise briefly introduces the concept of the cognitive miser, explaining that conserving energy and not thinking about things too much is a perfectly understandable tendency given our evolutionary past.

The Cognitive Miser: Ways to Avoid Thinking discusses the cognitive miser further, starting with the "Jack is looking at Anne but Anne is looking at George" problem, noting that one could arrive at the correct answer via disjunctive reasoning ("either Anne is married, in which case the answer is yes, or Anne is unmarried, in which case the answer is also yes") but most people won't bother. It then discusses attribute substitution (instead of directly evaluating X, consider the correlated and easier to evaluate quality Y), vividness/salience/accessibility effects, anchoring effects and the recognition heuristic. Stanovich emphasizes that he does not say that heuristics are always bad, but rather that one shouldn't always rely on them.

Framing and the Cognitive Miser extensively discusses various framing effects, and at the end notes that high-IQ people are not usually any more likely to avoid producing inconsistent answers to various framings unless they are specifically instructed to try to be consistent. This is mentioned to be a general phenomenon: if intelligent people have to notice themselves that an issue of rationality is involved, they do little better than their counterparts of lower intelligence.

Myside Processing: Heads I Win - Tails I Win Too! discusses "myside bias", people evaluating situations in terms of their own perspective. Americans will provide much stronger support for the USA banning an unsafe German car than for Germany banning an unsafe American car. People will much more easily pick up on inconsistencies in the actions of their political opponents than the politicians they support. They will also be generally overconfident, be appalled at others exhibiting the same unsafe behaviors they themselves exhibit, underestimate the degree to which biases influence our own thinking, and assume people understand their messages better than they actually do. The end of the chapter surveys research on the linkage between intelligence and the tendency to fall prey to these biases. It notes that intelligent people again do moderately better, but only when specifically instructed to avoid bias.

A Different Pitfall of the Cognitive Miser: Thinking a Lot, but Losing takes up the problem of failing to override your autonomous processing even when it would be necessary. Most of this chapter is covered by my previous discussion of override failures in the Cognitive Miser post.

Mindware Gaps introduces in more detail a different failure mode: that of mindware gaps. It also introduces and explains the concepts of Bayes' theorem, falsifiability, base rates and the conjunction error as crucial mindware for avoiding many failures of rationality. It notes that thinking dispositions for actually actively analyzing things could be called "strategic mindware". The chapter concludes by noting that the useful mindware discussed in the chapter is not widely and systematically taught, leaving even intelligent people gaps in their mindware that makes them subject to failures of rationality.

I mostly covered the contents of Contaminated Mindware in my post about mindware problems.

How Many Ways Can Thinking Go Wrong? A Taxonomy of Irrational Thinking Tendencies and Their Relation to Intelligence summarizes the content of the previous chapters and organizes the various biases into a taxonomy of biases that has the main categories of the Cognitive Miser, Mindware Problems, and Mr. Spock Syndrome. I did not previously cover Mr. Spock Syndrome because as Stanovich says, it is not a fully cognitive category. People with the syndrome have a reduced ability to feel emotions, which messes up their ability to behave appropriately in various situations even though their intelligence remains intact. Stanovich notes that the syndrome is most obvious with people who have suffered severe brain damage, but difficulties of emotional regulation and awareness do seem to also correlate negatively with some tests of rationality, as well as positive life outcomes, even when intelligence is controlled for.

The Social Benefits of Increasing Human Rationality - and Meliorating Irrationality concludes the book by arguing that while increasing the average intelligence of people would have only small if any effects on general well-being, we could reap vast social benefits if we actually tried to make people more rational. There's evidence that rationality would be much more malleable than intelligence. Disjunctive reasoning, the tendency to consider all possible states of the world when deciding among options, is noted to be a rational thinking skill of high generality that can be taught. There also don't seem to be strong intelligence-related limitations on the ability to think disjunctively. Much other useful mindware, like that of scientific and probabilistic reasoning. While these might be challenging to people with a lower IQ, techniques such as implementation intention may be easier to learn.

An implementation intention is formed when the individual marks the cue-action sequence with the conscious, verbal declaration of "when X occurs, I will do Y." Often with the aid of the context-fixing properties of language, the triggering of this cue-action sequence on just a few occasions is enough to establish it in the autonomous mind. Finally, research has shown that an even more minimalist cognitive strategy of forming mental goals (whether or not they have implementation intentions) can be efficacious. For example, people perform better at a task when they are told to form a mental goal ("set a specific, challenging goal for yourself") for their performance as opposed to being given the generic motivational instructions ("do your best").

Stanovich also argues in favor of libertarian paternalism: shaping the environment so that people are still free to choose what they want, but so that the default choice is generally the best one. For instance, countries with an opt-out policy for organ donation have far more donors than the countries with an opt-in policy. This is not because the people in one country would be any more or less selfish than those in other countries, but because people in general tend to go with the default option. He also argues that it would be perfectly possible though expensive to develop general rationality tests that would be akin to intelligence tests, and that also using RQ proxies for things such as college admission would have great social benefits.

In studies cited in this book, it has been shown that:
  • Psychologists have found ways of presenting statistical information so that we can make more rational decisions related to medical matters and in any situation where statistics are involved.
  • Cognitive psychologists have shown that a few simple changes in presenting information in accord with default biases could vastly increase the frequency of organ donations, thus saving thousands of lives.
  • Americans annually pay millions of dollars for advice on how to invest their money in the stock market, when following a few simple principles from decision theory would lead to returns on their investments superior to any of this advice. These principles would help people avoid the cognitive biases that lead them to reduce their returns - overreacting to chance events, overconfidence, wishful thinking, hindsight bias, misunderstanding of probability.
  • Decision scientists have found that people are extremely poor at assessing environmental risks. This is mainly because vividness biases dominate people's judgment to an inordinate extent. People could improve, and this would make a huge difference because these poor assessments come to affect public policy (causing policy makers to implement policy A, which saves one life for each $3.2 million spent, instead of policy B, which would have saved one life for every $220,000 spent, for example).
  • Psychologists from various specialty areas are beginning to pinpoint the cognitive illusions that sustain pathological gambling behavior - pseudodiagnosticity, belief perseverance, over-reacting to chance events, cognitive impulsivity, misunderstanding probability - behavior that destroys thousands of lives each year.
  • Cognitive psychologists have studied the overconfidence effect in human judgment - that people miscalibrate their future performance, usually by making overoptimistic predictions. Psychologists have studied ways to help people avoid these problems in self-monitoring, making it easier for people to plan for the future (overconfident people get more unpleasant surprises).
  • Social psychological research has found that controlling the explosion of choices in our lives is one of the keys to happiness - that constraining choice often makes people happier.
  • Simple changes in the way that pension plans are organized and administered could make retirement more comfortable for millions of people.
  • Probabilistic reasoning is perhaps the most studied topic in the decision-making field, and many of the cognitive reforms that have been examined - for example, eliminating base-rate neglect - could improve practices in courtrooms, where poor thinking about probabilities have been shown to impede justice.

These are just a small sampling of the teachable reasoning strategies and environmental fixes that could make a difference in people's lives, and they are more related to rationality than intelligence. They are examples of the types of outcomes that would result if we all became more rational thinkers and decision makers. They are the types of outcomes that would be multiplied if schools, businesses, and government focused on the parts of cognition that intelligence tests miss. Instead, we continue to pay far more attention to intelligence than to rational thinking. It is as if intelligence has become totemic in our culture, and we choose to pursue it rather than the reasoning strategies that could transform our world.

New to LessWrong?

New Comment
55 comments, sorted by Click to highlight new comments since: Today at 5:34 PM

I can't remember where I read this:

"Whenever people say 'I'm not book smart, but I am street smart,' I hear 'I'm not real smart, but I am imaginary smart.'"

Or the other way around - sometimes the 'book smart - street smart' is a put down of the geek as well, not just used against the non-geek.

People said of my brother 'How can someone so smart be so dumb?'. He was both book smart and street smart. In fact he lived to 60 without having a wife, a house, a car, a permanent job, more belongings then he could carry and so on. He was never an alcoholic, addict, gambler or anything like that. He spent his life as a nomad and came to no harm. He knew his streets. He was almost saintly in the way he treated others. I never heard him say something unkind in his whole life. He had a very high IQ and knew a great deal fact-wise. His memory was good. He played an excellent game of chess. What he didn't have was any sense of perspective or way of making plans that would work out or way of accurately judging others and so on. He had no way of using his book smarts or his street smarts. I have often wondered if there was a word for what his was missing in his makeup. Anyone have a word?

Two words: He had no worldly ambition.

[missing the point]
That was five words.
[/missing the point]

I like the implication that if you'd missed off the second set of tags, everything you wrote from now on on LW would be irrelevant.

Whoops, I think you forgot your closing ta-... Ah! I get it now. I should've closed that tag years ago.

That's not particularly well-formed, is it now?

I'm very strongly tempted to respond with a "Yo mama" joke here.

Not to worry, a user-agent will usually add closing tags as needed, except where it's ambiguous. Most likely the error would not have proceeded past the end of the current block-level element.

Of course, that's assuming those bbcode-style tags are being rendered straight into html...

Presumably the people who said those things about your brother might have said similar things about Socrates, Buddha and Jesus. All three of those personages have had their share of critics during their lifetimes. Few of those critics' names or ideas (to say nothing of their wives, houses and belongings) survive to the present day. The few that did are mostly preserved and presented as negative examples and antagonists.

Based on the description provided, I am not sure I would say there was something missing in your brother's make-up; no major desirable traits, anyhow. Have you ever considered that your brother's sense of perspective may have far surpassed your own?

As to words, how about 'ascetic'?

Well actually no. If he had had a sense of perspective then I think he might well have been ascetic. Of course, being my brother I had a lot of respect and love for him, but I have to be realistic about his abilities.

I'm not sure exactly what you mean by "he didn't have ... any sense of perspective or way of making plans that would work out or way of accurately judging others". Did he have a form of high-functioning autism or Asperger's?

Yes I have thought that Asperger's was a possibility but along with something else. In any case I seems to me that book smarts and street smarts does not cover the spectrum of what we call intelligence. It is like, as an analogy, someone knows how to add and how to multiple but has no idea when it is appropriate to add and when it is appropriate to multiple.

Meh - to me, that sounds like a cheap dismissal of the "street smarts" of non-geeks.

"If you haven't memorized the first hundred digits of Pi, you're a moron"

Probably, but I think it has salience in the original-post context of George W. Bush. I'm not entirely sure how the son of a career politician oil magnate could be non-ironically described as having 'street'-anything.

He might not have "street smarts" but his people skills were good enough to get him the nomination.

He clearly has some smarts of some denomination. My point is that he's successfully graduated from one of the best educations money can buy. If apologists feel the need to draw attention to his 'street smarts' then something, somewhere, has gone a bit wrong.

Was there some sort of prioritization of what bias needs to be attacked hardest to spare us the most damage?

I would speculate when the investigations are finished we will find the BP spill was caused by human error; the biases displayed in making the bad decision were foremost either herd instincts, not wanting to rock the boat, or some very similar cognitively distorted error made by a group of people who all have very high IQ's.

I also think George Bush 2 is a poor example. He is a performer and part of his schtick is not being an uppity intellectual wanker.

Was there some sort of prioritization of what bias needs to be attacked hardest to spare us the most damage?

Not really.

Your reviews have convinced me to read this book in hopes of using it to introduce people to rationality.

He also argues that it would be perfectly possible though expensive to prevent general rationality tests that would be akin to intelligence tests, and that also using RQ proxies for things such as college admission would have great social benefits.

Is there a typo here? The two clauses don't seem to quite go together.

I didn't understand that either, maybe "prevent" should be "present"?

I think he meant to write "invent".

Changed it to "develop". (I think "invent" was what I originally meant, but "develop" makes more sense.)

Thank you for a very interesting sequence. I've been trying to find a next step for someone interested in rationality and currently going through the HP fanfic that Eliezer is writing, and this book just might be it.

I am also developing a slight interest in measuring rationality. Does the book share any insights on that?

I am also developing a slight interest in measuring rationality. Does the book share any insights on that?

Mostly it just mentions various experimental setups where people often end up giving inconsistent or irrational answers, such as the two ways of framing taxation thing. You could consider those tests of rationality. The type of personality tests that measure your thinking dispositions also seem to give results that are correlated with rationality.

I'm wondering if Bush's IQ was measured before or after his years of alcohol abuse...

Before, of course; adults rarely take IQ tests, though it would be interesting to know his GMAT scores [actually, those were only another 5 years later]. The two known scores are his SATs and his military tests. As Sailer (last link) always says,

the only election Bush ever lost was a 1978 Congressional race in the Texas Panhandle, where his opponent made fun of Bush for having degrees from Yale and Harvard.

Bush resolved never to get out-dumbed again.

ETA: my impression, probably from [the video James Andrix cites below], is that Bush sounded much more coherent during the gubernatorial elections than during the presidential ones, which would rule out a direct effect of alcohol.

I've also heard that Bush is a better speaker in his native Texas accent / dialect (which he didn't use when campaigning nationally).

That's an interesting video. I'd heard of his great performance during gubernatorial debates before, but never seen the evidence myself. One thing to be cautious about here is confirmation bias, like how the clips were selected...

The other thing is, the original theory I heard this evidence raised to defend was that Bush was intentionally appearing dumb in order to get elected, which went along with another use for appearing dumb in international strategic contexts so that long term strategic power grabs could be written off as being caused by any number of other things. For Iraq, someone might think "He just wanted to finish what his dad started" and stop thinking right there, because they expect no deeper motives.

The video you linked to suggests that its not a poker face, but is instead a medical condition.

I think we should be able to figure out the difference based on the post-presidential verbal performance - if he gets better, then he was probably faking for strategic reasons (and doesn't care if people know it by now), but if he gets worse, then its more likely to be a long term medical condition.

The guy who runs Stratfor ("Economic, political and military strategic forecasting") opined that the main reason for the invasion of Iraq was to put pressure on Saudi Arabia to stop funding religious schools that produce most of the Islamic extremists, direct military action against Saudi Arabia having been viewed as too costly.

I think the main reason for the invasion of Iraq was the unprecedented influence of the Project for the New American Century. http://en.wikipedia.org/wiki/Project_for_the_New_American_Century

I think that's basically a category error. Yes, you need to know who is making decisions before you can assess why they made them, but it certainly doesn't answer the question of why they made the decisions.

Also, I don't see any puppet strings. Why talk about PNAC, rather than Team B? Rumsfeld and Wolfowitz (but not Cheney) were paranoid decades ago.

Could it have been the convergence of three factors: money to be made from a war, Iraq weakened by sanctions, and the shock of 9/11?

By talking about the PNAC, I mean both that the war was caused by influential member individuals of the PNAC, as well as because of the idealistic cause behind the PNAC -- continued American hegemony. The Iraq War was a failed attempt at continuing American hegemony into the 21st century.

[-][anonymous]14y20

widely and systematically thought

-> widely and systematically taught?

Yes, thank you.

Re: "Stanovich notes that if we used a broad definition of intelligence, where intelligence only meant acting in an optimal manner, then this expression wouldn't make any sense."

Definitions along those lines are often used by machine intelligence researchers. There, definining intelligence as the principle factor across a broad range of human cognitive abilities makes little sense.

Typo: "Why Intelligence People Doing Foolish Things Is No Surprise"? Edit: Nevermind, that's the actual title of a chapter.

That was a typo, actually. Thanks.

(I'm guessing there's a missing word in "but that doesn't [negate?] the fact that people overvalue it".)

Yeah, there was. Fixed, thanks.

I'm very much against this view of biases. They're very much rational and help us in the real world all the time.

It's a better question to ask, how to best employ them.

Like say, using confirmation bias to reinforce ideas that empiricism, experience & practice and the scientific method are both great things.

I take it Stanovich is doing a lot of experiments where he controls for IQ, or compares performance within and across IQ groups. Here is my concern... there is always measurement error, and the more error in his measure of IQ, the more it will appear he's measuring something distinct from IQ which he terms "rationality."

That said, I also agree that IQ, and G, are often reified. The point is, I'm not sure Stanovich has succeeded in carving cognition skills at their joints, but I don't have anything better to offer.

Ah, now this is a lovely summary of the book. I agree that we should find ways to improve the general level of rationality.

Also, how much more rational do you all think you are after having been on LW for some time?

I think LW has done a lot more to introduce a lot of new people to existing material than it has done to push the frontiers.

I consider myself in the middle of the bell curve when it comes to intelligence. I'm average. I am, however, more rational than some very smart people I know..

This sounds like "I am not very good looking, but have a great sense of humor." ;)

: )

More like: I'm not stunning, just ordinary good looking, and I also have a great sense of humor.

I didn't say I was stupid, just not a genius.

You missed my point. (Which you would not have, if you had read The Upside of Irrationality by Dan Ariely, an excellent book.)

Hm, isn't he saying that unattractive people claim they value attributes other than physical attractiveness when selecting a mate? If that's what you're referring to, hilarious. If not, speak more slowly next time. : )

That's true.

But the parallel was a bit more specific: "Good sense of humor" (which he concretely brought up as the most typical example) is an attribute one can easily claim to have as it is impossible to measure.