Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.


20 Post author: Eliezer_Yudkowsky 30 September 2007 07:29PM

Followup toThe Bottom Line, What Evidence Filtered Evidence?

In "The Bottom Line", I presented the dilemma of two boxes only one of which contains a diamond, with various signs and portents as evidence.  I dichotomized the curious inquirer and the clever arguer.  The curious inquirer writes down all the signs and portents, and processes them, and finally writes down "Therefore, I estimate an 85% probability that box B contains the diamond."  The clever arguer works for the highest bidder, and begins by writing, "Therefore, box B contains the diamond", and then selects favorable signs and portents to list on the lines above.

The first procedure is rationality.  The second procedure is generally known as "rationalization".

"Rationalization."  What a curious term.  I would call it a wrong word.  You cannot "rationalize" what is not already rational.  It is as if "lying" were called "truthization".

On a purely computational level, there is a rather large difference between:

  1. Starting from evidence, and then crunching probability flows, in order to output a probable conclusion.  (Writing down all the signs and portents, and then flowing forward to a probability on the bottom line which depends on those signs and portents.)
  2. Starting from a conclusion, and then crunching probability flows, in order to output evidence apparently favoring that conclusion. (Writing down the bottom line, and then flowing backward to select signs and portents for presentation on the lines above.)

What fool devised such confusingly similar words, "rationality" and "rationalization", to describe such extraordinarily different mental processes?  I would prefer terms that made the algorithmic difference obvious, like "rationality" versus "giant sucking cognitive black hole".

Not every change is an improvement, but every improvement is necessarily a change.  You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true. To improve our beliefs, we must necessarily change our beliefs. Rationality is the operation that we use to obtain more truth-value for our beliefs by changing them.  Rationalization operates to fix beliefs in place; it would be better named "anti-rationality", both for its pragmatic results and for its reversed algorithm.

"Rationality" is the forward flow that gathers evidence, weighs it, and outputs a conclusion.  The curious inquirer used a forward-flow algorithm: first gathering the evidence, writing down a list of all visible signs and portents, which they then processed forward to obtain a previously unknown probability for the box containing the diamond.  During the entire time that the rationality-process was running forward, the curious inquirer did not yet know their destination, which was why they were curious.  In the Way of Bayes, the prior probability equals the expected posterior probability:  If you know your destination, you are already there.

"Rationalization" is a backward flow from conclusion to selected evidence.  First you write down the bottom line, which is known and fixed; the purpose of your processing is to find out which arguments you should write down on the lines above.  This, not the bottom line, is the variable unknown to the running process.

I fear that Traditional Rationality does not properly sensitize its users to the difference between forward flow and backward flow.  In Traditional Rationality, there is nothing wrong with the scientist who arrives at a pet hypothesis and then sets out to find an experiment that proves it.  A Traditional Rationalist would look at this approvingly, and say, "This pride is the engine that drives Science forward."  Well, it is the engine that drives Science forward.  It is easier to find a prosecutor and defender biased in opposite directions, than to find a single unbiased human.

But just because everyone does something, doesn't make it okay.  It would be better yet if the scientist, arriving at a pet hypothesis, set out to test that hypothesis for the sake of curiosity—creating experiments that would drive their own beliefs in an unknown direction.

If you genuinely don't know where you are going, you will probably feel quite curious about it.  Curiosity is the first virtue, without which your questioning will be purposeless and your skills without direction.

Feel the flow of the Force, and make sure it isn't flowing backwards.


Part of the Against Rationalization subsequence of How To Actually Change Your Mind

Next post: "A Rational Argument"

Previous post: "What Evidence Filtered Evidence?"

Comments (22)

Sort By: Old
Comment author: Robin_Hanson2 30 September 2007 08:10:17PM 24 points [-]

Sadly, I almost always surprise economics graduate students looking for topics to research when I ask them; "What question, where you do not know the answer, would you most like to answer?"

Comment author: Naadir_Jeewa 30 September 2007 08:47:43PM 0 points [-]

How would this relate to *shock* Bruno Latour's conceptualization of Actor-Network-Theory, where the sociologist simply tries to maximise the number of sources of uncertainty in a set of trials, without resorting to a "explanatory social theory"?

Comment author: Adirian 30 September 2007 09:05:18PM 1 point [-]

I find the linguistic distinction to be better than you relate - to rationalize something is to start with something that isn't rational. (As if it were rational, it wouldn't need to be rationalized - it's already there.)

That being said, rationalization in action isn't always bad, because we don't always have conscious understanding of the algorithm used to produce our conclusions. This would be like, to use your example, Einstein coming to the conclusion of relativity - and then attempting to understand how he got there. Rationalization in this case is a useful tool, as it is, in effect, an attempt to obtain the variables that originally went into the algorithm, perhaps to examine their validity.

If you already understand how you got to a conclusion which you are then attempting to bolster - if the evidence that is filtering evidence is being ignored - then it is precisely as bad as you say.

Comment author: adirian2 30 September 2007 09:06:46PM 0 points [-]

I apologize, didn't mean to double post.

Comment author: pdf23ds 30 September 2007 09:58:12PM 4 points [-]

It is as if "lying" were called "truthization".

Apologies for the content-free comment, but this is a really great line. Worthy of Stephen Colbert.

Comment author: pdf23ds 30 September 2007 10:04:01PM 1 point [-]

Of course, in an etymological sense, "rationalization" doesn't seem so odd. "Reason" means both logic and motivation. Those two concepts are conflated in the word and related words, and "rationalization" is simply formed from "rationale". (Actual etymologists, or users of Google, may feel free to correct me.)

Comment author: Vladimir_Nesov 01 October 2007 01:20:33PM 1 point [-]

I agree with Adirian. Rationalization is a process of rational-explanation-seeking. It starts from statement that was obtained by non-rational process (as when you overheard something, or intuitively guessed something) and then creates a rational explanation according to one's concept of rationality, concurrently adjusting statement if necessary. So normal rationalization does change the conclusion: it can change its status from 'suspicious statement' to 'belief', or it can adjust it to be consistent with facts. Now biased rationalization uses 'biased rationality' according to which it builds explanation, for example that 'clever arguer' applies selection bias.

Comment author: Eliezer_Yudkowsky 01 October 2007 03:36:24PM 0 points [-]

It starts from statement that was obtained by non-rational process (as when you overheard something, or intuitively guessed something)

An intuitive guess is non-scientific but not non-rational.

Comment author: Doug_S. 01 October 2007 05:16:11PM 3 points [-]

Random comment:

Many years ago, there were a series of articles written by the pseudonym Archibald Putt, collectively referred to as "Putt's Laws", that appeared in Research/Development magazine. One law is relevant to the topic at hand.

"Decisions are justified by benefits to the organization; they are made by considering benefits to the decisionmakers."

If it is easier to lie convincingly when you believe the lie, then rationalization makes perfect sense. One makes a decision based on selfish, primarily unconscious motives, and then comes up with a semi-convincing rationalization for public consumption. "I stole that because I deserved it" would be a classic example of this kind of justification.

Comment author: Vladimir_Nesov2 01 October 2007 05:37:04PM 0 points [-]

Eliezer: An intuitive guess is non-scientific but not non-rational

It doesn't affect my point; but do you argue that intuitive reasoning can be made free of bias?

Comment author: [deleted] 02 January 2013 09:13:37PM 0 points [-]

An intuitive guess can be made without biasing the result (accept or reject), so long as one does not privilege the hypothesis.

Comment author: Vag 01 July 2010 08:10:32AM 0 points [-]

Your wonderful essay contains a flaw.

"box B contains the diamond"

There is no way in reality to check correctness of reasoning result "directly" (unable to "open box and see if it contains brilliant"). But, if result of reasoning is not directly influences the reasoner, it is also unfeasible .

So, correct story is: "one of two melted unopenable boxes contains bomb with timer. The task is select one box and throw in deep well, or else it shall explode and mutilate the reasoner"

Comment author: MoreOn 26 December 2010 07:32:51AM *  1 point [-]

Try answering this without any rationalization:

In my middle school science lab, a thermometer showed me that water boiled at 99.5 degrees C and not 100. Why?

Comment author: datadataeverywhere 26 December 2010 07:51:14AM 11 points [-]

I suspect you have a point that I'm missing.

My take is: either the reading was wrong (experimental error of some kind), or it wasn't wrong. If it wasn't wrong, then your water was boiling at a 99.5 degrees. There are a number of plausible explanations for the latter; the one that I assign the highest prior to is that you were at an elevation higher than sea level.

So, my answer is in the form of a probability distribution. Give me more evidence, and I will refine it, or demand and answer now, and I will tell you "altitude", my current most plausible candidate (experimental error is my second candidate, first with how (where in the water) you measured, then with the quality of the thermometer. After that trails things like impurities in the water).

Comment author: ksvanhorn 18 January 2011 02:55:14AM 3 points [-]

What altitude were you at?

Comment author: Torvaun 12 February 2011 04:12:07PM 3 points [-]

My experience leads me to assume that the thermometer was mismarked. My high school chemistry teacher drilled into us that the thermometers we had were all precise, but of varying accuracy. A thermometer might say that water boils at 99.5 C, but if it did, it would also say that it froze at -0.5 C. Again, there are conditions that actually change the temperature at which water boils, so it's possible you were at a lower atmospheric pressure or that the water was contaminated. But, given that we have a grand total of one data point, I can't narrow it down to a single answer.

Comment author: Desrtopa 12 February 2011 05:28:59PM 1 point [-]

What elevation was your school at?

Comment author: jslocum 20 March 2011 05:38:11PM *  8 points [-]

You've missed a key point, which is that rationalization refers to a process in which one of many possible hypothesis is arbitrarily selected, which the rationalizer then attempts to support using a fabricated argument. In your query, you are asking that a piece of data be explained. In the first case, one filters the evidence, rejecting any data that too strongly opposes a pre-selected hypothesis. In the second case, one generates a space of hypothesis that all fit the data, and selects the most likely one as a guess. The difference is between choosing data to fit a hypothesis, and finding a hypothesis that best fits the data. Rationalization is pointing to a blank spot on your map and saying, "There must be a lake somewhere around there, because there aren't any other lakes nearby," while ignoring the fact that it's hot and there's sand everywhere.

Comment author: Origin64 05 November 2012 07:48:48PM *  0 points [-]

Not every change is an improvement, but every improvement is necessarily a change. You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true. To improve our beliefs, we must necessarily change our beliefs.

I know this of course, but the way you state it here really drives the point home. Well written.