Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Why truth? And...

45 Post author: Eliezer_Yudkowsky 27 November 2006 01:49AM

Some of the comments in this blog have touched on the question of why we ought to seek truth.  (Thankfully not many have questioned what truth is.)  Our shaping motivation for configuring our thoughts to rationality, which determines whether a given configuration is "good" or "bad", comes from whyever we wanted to find truth in the first place.

It is written:  "The first virtue is curiosity."  Curiosity is one reason to seek truth, and it may not be the only one, but it has a special and admirable purity.  If your motive is curiosity, you will assign priority to questions according to how the questions, themselves, tickle your personal aesthetic sense.  A trickier challenge, with a greater probability of failure, may be worth more effort than a simpler one, just because it is more fun.

Some people, I suspect, may object that curiosity is an emotion and is therefore "not rational". I label an emotion as "not rational" if it rests on mistaken beliefs, or rather, on irrational epistemic conduct: "If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm." Conversely, then, an emotion which is evoked by correct beliefs or epistemically rational thinking is a "rational emotion"; and this has the advantage of letting us regard calm as an emotional state, rather than a privileged default. When people think of "emotion" and "rationality" as opposed, I suspect that they are really thinking of System 1 and System 2—fast perceptual judgments versus slow deliberative judgments. Deliberative judgments aren't always true, and perceptual judgments aren't always false; so it is very important to distinguish that dichotomy from "rationality". Both systems can serve the goal of truth, or defeat it, according to how they are used.

Besides sheer emotional curiosity, what other motives are there for desiring truth? Well, you might want to accomplish some specific real-world goal, like building an airplane, and therefore you need to know some specific truth about aerodynamics. Or more mundanely, you want chocolate milk, and therefore you want to know whether the local grocery has chocolate milk, so you can choose whether to walk there or somewhere else. If this is the reason you want truth, then the priority you assign to your questions will reflect the expected utility of their information—how much the possible answers influence your choices, how much your choices matter, and how much you expect to find an answer that changes your choice from its default.

To seek truth merely for its instrumental value may seem impure—should we not desire the truth for its own sake?—but such investigations are extremely important because they create an outside criterion of verification: if your airplane drops out of the sky, or if you get to the store and find no chocolate milk, it's a hint that you did something wrong.  You get back feedback on which modes of thinking work, and which don't.  Pure curiosity is a wonderful thing, but it may not linger too long on verifying its answers, once the attractive mystery is gone.  Curiosity, as a human emotion, has been around since long before the ancient Greeks.  But what set humanity firmly on the path of Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world.  As far as sheer curiosity goes, spinning campfire tales of gods and heroes satisfied that desire just as well, and no one realized that anything was wrong with that.

Are there motives for seeking truth besides curiosity and pragmatism?  The third reason that I can think of is morality:  You believe that to seek the truth is noble and important and worthwhile.  Though such an ideal also attaches an intrinsic value to truth, it's a very different state of mind from curiosity.  Being curious about what's behind the curtain doesn't feel the same as believing that you have a moral duty to look there.  In the latter state of mind, you are a lot more likely to believe that someone else should look behind the curtain, too, or castigate them if they deliberately close their eyes.  For this reason, I would also label as "morality" the belief that truthseeking is pragmatically important to society, and therefore is incumbent as a duty upon all.  Your priorities, under this motivation, will be determined by your ideals about which truths are most important (not most useful or most intriguing); or your moral ideals about when, under what circumstances, the duty to seek truth is at its strongest.

I tend to be suspicious of morality as a motivation for rationality, not because I reject the moral ideal, but because it invites certain kinds of trouble.  It is too easy to acquire, as learned moral duties, modes of thinking that are dreadful missteps in the dance.  Consider Mr. Spock of Star Trek, a naive archetype of rationality.  Spock's emotional state is always set to "calm", even when wildly inappropriate.  He often gives many significant digits for probabilities that are grossly uncalibrated.  (E.g:  "Captain, if you steer the Enterprise directly into that black hole, our probability of surviving is only 2.234%"  Yet nine times out of ten the Enterprise is not destroyed.  What kind of tragic fool gives four significant digits for a figure that is off by two orders of magnitude?)  Yet this popular image is how many people conceive of the duty to be "rational"—small wonder that they do not embrace it wholeheartedly.  To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom.  People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake.

And yet if we're going to improve our skills of rationality, go beyond the standards of performance set by hunter-gatherers, we'll need deliberate beliefs about how to think with propriety.  When we write new mental programs for ourselves, they start out in System 2, the deliberate system, and are only slowly—if ever—trained into the neural circuitry that underlies System 1.  So if there are certain kinds of thinking that we find we want to avoid—like, say, biases—it will end up represented, within System 2, as an injunction not to think that way; a professed duty of avoidance.

If we want the truth, we can most effectively obtain it by thinking in certain ways, rather than others; and these are the techniques of rationality.  Some of the techniques of rationality involve overcoming a certain class of obstacles, the biases...

(Continued in next post:  "What's a bias, again?")

 

Part of the sequence Map and Territory

Next post: "What is Evidence?"

 

Comments (33)

Sort By: Old
Comment author: Doug_S. 30 January 2008 09:18:27PM 5 points [-]

Yet nine times out of ten the Enterprise is not destroyed. What kind of tragic fool gives four significant digits for a figure that is off by two orders of magnitude?

One who doesn't understand the Million To One Chance principle that operates in fictional universes. If the Star Trek universe didn't follow the laws of fiction, the Enterprise would have been blown up long ago. ;)

See also: Straw Vulcan

Comment author: MrCheeze 31 January 2011 03:49:20AM 18 points [-]

Maybe in ninety-eight universes out of 100 it does blow up and we just see the one that's left; and he's actually giving an accurate number. :P

Comment author: shokwave 31 January 2011 01:00:02PM 34 points [-]

The TV show version of the anthropic principle: all the episodes where the Enterprise does blow up aren't made.

Comment author: Raemon 23 November 2011 06:52:14AM 7 points [-]

Except one.

Comment author: Alex2 12 September 2008 05:59:03PM 1 point [-]

Does this sentence contain a typo?

"If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, the Way opposes your calm."

Comment author: Eliezer_Yudkowsky 12 September 2008 06:25:26PM 0 points [-]

Fixed.

Comment author: RickJS 24 May 2010 01:24:32AM *  9 points [-]

Thanks, Eliezer!

“Are there motives for seeking truth besides curiosity and pragmatism?”

I can think of several that have showed up in my life. I’m offering these for consideration, but not claiming these are good or bad, pure or impure etc. Some will doubtless overlap somewhat with each other and the ones stated.

  1. As a weapon. Use it to win arguments (sometimes the point of an argument is to WIN, never mind learning the truth. I've got automatic competitiveness I need to keep on a short leash). Use it to win bar room bets. Acquire knowledge about the “buttons” people have, and use it to manipulate them. Use it to thwart opposition to my plans, however sleazy. (“What are we going to do tonight, Brain?” ... )
  2. As evidence that I deserve an A in school. Even if I never have a pragmatic use for the knowledge, there is (briefly) value in demonstrably having the knowledge.
  3. As culture. I don’t think I have ever found a practical use for the facts of history ( of science, of politics, or of art ), but they participated in shaping my whole world view. Out of that, I came out of retirement and dedicated myself to saving humanity. Go figure.
  4. As a contact, as in, “I know Nick Bostrom.” (OK, that’s a bit of a stretch, but it is partly informational.) 5, As pleasure & procreation, as in, “Cain knew his wife.” ;-)

“To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom. People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake.” Yes. I say, “Morality is for agents that can’t figure out the probable consequences of their actions.” Which includes me, of course. However, whenever I can make a good estimate, I pretty much become a consequentialist.

Seeking knowledge has, for me, an indirect but huge value. I say: Humanity needs help to survive this century, needs a LOT of help. I think Friendly AI is our best shot at getting it. And we’re missing pieces of knowledge. There may be whole fields of knowledge that we’re missing and we don’t know what they are.

I would not recommend avoiding lines of research that might enable making terribly powerful weapons. We’ve already got that problem, there’s no avoiding it. But there’s no telling what investigations will produce bits of information that will trigger some human mind into a century-class breakthrough that we had no idea we needed.

Comment author: langej 14 July 2010 11:56:21AM -1 points [-]

The significant digit anecdote reminds me: why does the Dow Jones giver their average with 2 decimal points?

I do have a couple of problems, though

1) It is written: "The first virtue is curiosity." - Written by whom? 2) …curiosity is an emotion… - says who? 3) To seek truth merely for its instrumental value may seem impure… – Why? To whom? 4) If we want the truth, we can most effectively obtain it by thinking in certain ways – and if you think the way I tell you to think, you’ll wind up with my truth

Comment author: RichardKennaway 14 July 2010 12:39:09PM 3 points [-]

It is written: "The first virtue is curiosity." - Written by whom?

By Eliezer.

Comment author: simplicio 16 August 2010 12:30:12AM *  4 points [-]

From TVtropes:

Star Trek The Original Series episode "This Side of Paradise". Mr. Spock has been affected by spores that release his emotional side. He and his love interest Leila Kalomi are looking at clouds.

Spock: That one looks like a dragon. You see the tail and the dorsal spines?

Leila: I've never seen a dragon.

Spock: I have. On Berengaria 7. But I've never stopped to look at clouds before. Or rainbows. I can tell you exactly why one appears in the sky, but considering its beauty has always been out of the question.

Sigh.

Comment author: JohnDavidBustard 01 September 2010 12:34:45PM 4 points [-]

Lets not forget, arguably the most important reason.

Because it makes us feel good.

We can feel superior to others, because we can do something that few other people can. We can collect instances where our approach is beneficial and use that to validate our self worth. And we can form a community that validates our strengths and ignores our weaknesses. All perfectly reasonable motivations (provided our satisfaction is a reasonable goal).

In my own field (Computer Vision), there are those who pursue it rationally (with rigorous mathematical analysis) and those who pursue it heuristically (creating a variety of systems and testing them on small samples). These approaches seem to mirror the determined search for truth and the pragmatic "go with what feels like it works" approaches. Without rigorously analysing them (although this may be possible) both approaches seem to deliver benefit with no clear winner in terms of delivering approaches that are practically applied or used as the basis for further work. I think it is interesting to apply this meta analysis to reason, i.e. can we scientifically determine whether approaching problems reasonably conveys advantage? Is there an optimal balance?

Comment author: RobinZ 01 September 2010 04:21:31PM 3 points [-]

By "most important reason" do you mean "most compelling justification" or "predominant cause"?

Comment author: JohnDavidBustard 01 September 2010 04:53:13PM 3 points [-]

I would suggest both, and I would add that I don't think this inherently diminishes the value of pursuing truth. I am increasingly of the belief that in order to be content it is necessary to pick ones community and embrace its values. What I love about this community is its willingness to question itself as much as the views of others. I think it's useful to acknowledge what we really enjoy and be hesitant of explanations that attribute objective value to enjoyable activities. Doing so risks erasing self doubt and can lead to the adoption of strong moral values that distort our lives to such an extent that they ultimately make us miserable.

Comment author: Nisan 01 September 2010 05:13:15PM 6 points [-]

there are those who pursue it rationally (with rigorous mathematical analysis) and those who pursue it heuristically (creating a variety of systems and testing them on small samples). [...] both approaches seem to deliver benefit with no clear winner

"Rationality" is what I would call the meta-analysis which concludes that both approaches are equally valid in this field.

Comment author: michaelcurzi 02 January 2011 01:03:08PM 0 points [-]

"For this reason, I would also label as "morality" the belief that truthseeking is pragmatically important to society, and therefore is incumbent as a duty upon all."

Morality doesn't need to have anything to do with society or duty. Consider the case of an rational ethical egoist, to whom acting in one's self-interest and for one's own values is virtuous.

Comment author: thomblake 12 May 2011 10:26:42PM 0 points [-]

Morality doesn't need to have anything to do with society or duty. Consider the case of an rational ethical egoist, to whom acting in one's self-interest and for one's own values is virtuous.

If that person is a human, and thinks that ethical egoism does not have anything to do with society or duty, then they are mistaken.

Comment author: michaelcurzi 13 May 2011 05:34:50PM 0 points [-]

Why?

Comment author: CuSithBell 13 May 2011 05:37:21PM 1 point [-]

I'd guess because humans often contain concepts of duty and the like, and have experiences vastly contingent on social / societal contexts.

Comment author: thomblake 13 May 2011 09:06:34PM 2 points [-]

Maintaining interpersonal relationships is vital to the human condition. As Aristotle put it, "The solitary life is perhaps suitable for a god or a beast, but not for a man". Friendships are a necessary part of flourishing for humans, and aside from that we are almost always in a context where our success depends upon our interactions with others.

Comment author: Alexandros 20 April 2011 07:22:50PM 3 points [-]

There is more discussion of this post here as part of the Rerunning the Sequences series.

Comment author: omeganaut 12 May 2011 03:24:46PM 0 points [-]

I'll be honest, I have a serious problem with hypocrites, and so I warn everyone I know if they start heading down that path. In your article, you say that morality is perhaps the most suspect method of rationality. Yet, you yourself, by putting up these articles and arguing that everyone should use rational thought, seem to have a moral motivation for rationality. I am not saying that this is your only motivation, but it seems to be the motivation behind these posts. However, I do appreciate that you respect morality by mentioning how important it is in pursuing paths that will not result in horrible consequences. I think that maybe you should allow yourself to admit that morality is a good motivator if used with other good types of motivations to seek truth.

Comment author: Arandur 13 August 2011 01:20:41AM 1 point [-]

Here's an interesting take on the "morality" side: It may be morally incumbent on some to look behind the curtain, and not for others. Since knowing about biases can hurt people, it may well be that those who are "fit" to look behind the curtain are in fact required to be the guardians of said curtain, forbidding anyone without the proper light and knowledge from looking behind it, but acting upon the knowledge gained for the benefit of society.

..... Hence, the Conspiracy.

Comment author: lowasser 05 May 2012 09:54:53PM 1 point [-]

I am trying to win an argument, and I am having trouble defeating the following claim:

It can, under certain scenarios, be instrumental (in the sense of achieving values) to believe in something which is false -- usually by virtue of a placebo effect. For example: believing you are more likely to get a job offer than you really are, so you are confident at the job interview.

The counterargument I want to make, in my head, is that if you have the ability to deceive yourself to that extent -- to make yourself believe something that is false -- then you have the ability to believe that you won't get the job interview, but pretend that you think you will. I don't feel like that's a very solid or reassuring argument, though.

Comment author: TimS 05 May 2012 11:38:56PM 1 point [-]

I think the best response to the argument for instrumentally useful false beliefs is to think a little about the causal mechanism. Surely it is not the case that Omega reads your minds, sees your false confidence, and orders you hired.

As you noted, a more plausible mechanism is that the false confidence causes changes in affect (i.e. appearing confident) that are beneficial for the task. Or perhaps false over-confidence cancels false under-confidence that would have caused anxiety that would be detrimental for the task.

Once the causal chain is examined, the next thing to ask is whether the beneficial intermediate effects can be caused by something other than false belief. If so, you have answered the claim you are responding to. If not, examining why one doesn't believe it possible needs to be examined.

Comment author: AspiringRationalist 21 June 2012 06:20:57AM 0 points [-]

You should also examine the costs of each method of achieving the intermediate effects. Even if there are other ways available, maybe self-deception is the easiest, and the costs of that particular incorrect belief are small.

Comment author: royf 28 May 2012 04:46:45AM *  1 point [-]

"If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm."

This quote conflates "true beliefs" and what we may call "correct beliefs". True beliefs are ones which assign high probability to the truth, i.e. the actual state of things. Correct beliefs are ones which follow from an agent's priors and observations. The former are objective, the latter subjective but not irrational. If the iron has been cool the last 107 times it has approached your face, but hot this 108th time, your belief that it is cool is correct but false (perhaps better terms are needed).

Also, a belief is not binary. You may be 99.8% sure that the iron is hot and still rationally fear it. A hot iron on your face is far more costly than a needless avoidance.

Comment author: royf 28 May 2012 04:53:28AM 0 points [-]

There's an interesting duality between morality as "the belief that truthseeking is pragmatically important to society" and morality as the result of social truthseeking, which is closer to the usual sense, or rather what the usual sense would ideally be. I'd like to see this explored further if anyone has a link in mind.

Comment author: aceofspades 27 June 2012 07:05:09PM 1 point [-]

I'm not convinced that this post actually says anything. If seeking the truth is useful for any specific reason, then people who see some benefit from it will do so and if it isn't useful then they won't. Actually writing this out has made me think both this post and my comment haven't really said much, but I think that's because this discussion is too abstract to have any real use/meaning. Ideas which are true/work will work, ideas that aren't won't, and that's all that needs to be said, never mind this business about rationality and truth and curiosity.

Comment author: thomblake 27 June 2012 07:25:04PM *  1 point [-]

that's all that needs to be said

Would that this were true.

Indeed, if that were all there was to it, nothing would need to be said at all, as that's a tautology. But people manage to fail at noticing when things do / don't work anyway, and false ideas stick around a very long time.

Comment author: aceofspades 05 July 2012 06:31:46PM 0 points [-]

I just find it very unlikely that the specifics of how this post is constructed have much of an effect on correcting this issue.

Comment author: CephasAtheos 22 December 2012 11:09:35PM *  0 points [-]

Ah, but the seeker needs to find out if the answer - the truth - is beneficial. You can't not know the truth and make a decision without knowing the answer. That's just guessing.

My friend argues that believing in an afterlife (i.e. religion) is beneficial for some people because it gives them a (patently false!) sense of "security". So why tell them it's wrong to believe such a thing?

My answer is a) the fact that there's no afterlife is the truth, as far as humans know (i.e. as far as the evidence - or lack of evidence - shows); and b) it's wrong to believe in such a falsehood - in the sense that most people with such a belief tend to be either less ethical/moral (because they'll fix up the imbalance 'later'), or irrationally over-moral or hyper-ethical because they don't want to risk their slot in eternity's gravy train. Either way, they act irrationally and abnormally, and for the wrong reasons!

I can't think of much in life that could be worse than that. What a horrible life!

Comment author: Mark_Friedenbach 22 June 2014 11:32:11PM 0 points [-]

It is instructive to review this essay after reading the sequence regarding metaethics, morlaity, and planning algorithms. It lets you receive a deeper insight into how "morality as a motivation" might have come about and what its flaws are.