Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Guardians of the Truth

31 Post author: Eliezer_Yudkowsky 15 December 2007 06:44PM

Followup toTsuyoku Naritai, Reversed Stupidity is not Intelligence

The criticism is sometimes leveled against rationalists:  "The Inquisition thought they had the truth!  Clearly this 'truth' business is dangerous."

There are many obvious responses, such as "If you think that possessing the truth would license you to torture and kill, you're making a mistake that has nothing to do with epistemology."  Or, "So that historical statement you just made about the Inquisition—is it true?"

Reversed stupidity is not intelligence:  "If your current computer stops working, you can't conclude that everything about the current system is wrong and that you need a new system without an AMD processor, an ATI video card... even though your current system has all these things and it doesn't work.  Maybe you just need a new power cord."  To arrive at a poor conclusion requires only one wrong step, not every step wrong.  The Inquisitors believed that 2 + 2 = 4, but that wasn't the source of their madness.  Maybe epistemological realism wasn't the problem either?

It does seem plausible that if the Inquisition had been made up of relativists, professing that nothing was true and nothing mattered, they would have mustered less enthusiasm for their torture.  They would also have had been less enthusiastic if lobotomized.  I think that's a fair analogy.

And yet... I think the Inquisition's attitude toward truth played a role.  The Inquisition believed that there was such a thing as truth, and that it was important; well, likewise Richard Feynman.  But the Inquisitors were not Truth-Seekers.  They were Truth-Guardians.

I once read an argument (can't find source) that a key component of a zeitgeist is whether it locates its ideals in its future or its past.  Nearly all cultures before the Enlightenment believed in a Fall from Grace—that things had once been perfect in the distant past, but then catastrophe had struck, and everything had slowly run downhill since then:

"In the age when life on Earth was full...  They loved each other and did not know that this was 'love of neighbor'. They deceived no one yet they did not know that they were 'men to be trusted'. They were reliable and did not know that this was 'good faith'. They lived freely together giving and taking, and did not know that they were generous. For this reason their deeds have not been narrated. They made no history."
        —The Way of Chuang Tzu, trans. Thomas Merton

The perfect age of the past, according to our best anthropological evidence, never existed.  But a culture that sees life running inexorably downward is very different from a culture in which you can reach unprecedented heights. 

(I say "culture", and not "society", because you can have more than one subculture in a society.)

You could say that the difference between e.g. Richard Feynman and the Inquisition was that the Inquisition believed they had truth, while Richard Feynman sought truth.  This isn't quite defensible, though, because there were undoubtedly some truths that Richard Feynman thought he had as well.  "The sky is blue," for example, or "2 + 2 = 4".

Yes, there are effectively certain truths of science.  General Relativity may be overturned by some future physics—albeit not in any way that predicts the Sun will orbit Jupiter; the new theory must steal the successful predictions of the old theory, not contradict them.  But evolutionary theory takes place on a higher level of organization than atoms, and nothing we discover about quarks is going to throw out Darwinism, or the cell theory of biology, or the atomic theory of chemistry, or a hundred other brilliant innovations whose truth is now established beyond reasonable doubt.

Are these "absolute truths"?  Not in the sense of possessing a probability of literally 1.0.  But they are cases where science basically thinks it's got the truth.

And yet scientists don't torture people who question the atomic theory of chemistry.  Why not?  Because they don't believe that certainty licenses torture?  Well, yes, that's the surface difference; but why don't scientists believe this?

Because chemistry asserts no supernatural penalty of eternal torture for disbelieving in the atomic theory of chemistry?  But again we recurse and ask the question, "Why?"  Why don't chemists believe that you go to hell if you disbelieve in the atomic theory?

Because journals won't publish your paper until you get a solid experimental observation of Hell?  But all too many scientists can suppress their skeptical reflex at will.  Why don't chemists have a private cult which argues that nonchemists go to hell, given that many are Christians anyway?

Questions like that don't have neat single-factor answers.  But I would argue that one of the factors has to do with assuming a defensive posture toward the truth, versus a productive posture toward the truth.

When you are the Guardian of the Truth, you've got nothing useful to contribute to the Truth but your guardianship of it.  When you're trying to win the Nobel Prize in chemistry by discovering the next benzene or buckyball, someone who challenges the atomic theory isn't so much a threat to your worldview as a waste of your time.

When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth.  If there's some way to pump against entropy, generate new true beliefs along with a little waste heat, that same pump can keep the truth alive without secret police.  In chemistry you can replicate experiments and see for yourself—and that keeps the precious truth alive without need of violence.

And it's not such a terrible threat if we make one mistake somewhere—end up believing a little untruth for a little while—because tomorrow we can recover the lost ground.

But this whole trick only works because the experimental method is a "criterion of goodness" which is not a mere "criterion of comparison".  Because experiments can recover the truth without need of authority, they can also override authority and create new true beliefs where none existed before.

Where there are criteria of goodness that are not criteria of comparison, there can exist changes which are improvements, rather than threats.  Where there are only criteria of comparison, where there's no way to move past authority, there's also no way to resolve a disagreement between authorities.  Except extermination.  The bigger guns win.

I don't mean to provide a grand overarching single-factor view of history.  I do mean to point out a deep psychological difference between seeing your grand cause in life as protecting, guarding, preserving, versus discovering, creating, improving.  Does the "up" direction of time point to the past or the future?  It's a distinction that shades everything, casts tendrils everywhere.

This is why I've always insisted, for example, that if you're going to start talking about "AI ethics", you had better be talking about how you are going to improve on the current situation using AI, rather than just keeping various things from going wrong.  Once you adopt criteria of mere comparison, you start losing track of your ideals—lose sight of wrong and right, and start seeing simply "different" and "same".

I would also argue that this basic psychological difference is one of the reasons why an academic field that stops making active progress tends to turn mean.  (At least by the refined standards of science.  Reputational assassination is tame by historical standards; most defensive-posture belief systems went for the real thing.)  If major shakeups don't arrive often enough to regularly promote young scientists based on merit rather than conformity, the field stops resisting the standard degeneration into authority.  When there's not many discoveries being made, there's nothing left to do all day but witch-hunt the heretics.

To get the best mental health benefits of the discover/create/improve posture, you've got to actually be making progress, not just hoping for it.


Part of the Death Spirals and the Cult Attractor subsequence of How To Actually Change Your Mind

Next post: "Guardians of the Gene Pool"

Previous post: "Every Cause Wants To Be A Cult"

Comments (56)

Sort By: Old
Comment author: Chris 15 December 2007 07:57:53PM 2 points [-]

It's amusing to see 'criterion of goodness' as a simile for 'criterion of correctness'. The Inquisition believed they were both 'correct' and 'good'. In torturing you, they were saving your soul, which was, for them, the ultimate in Utility. So, in calculating utility, beware of your assumptions.

Comment author: Nick_Tarleton 15 December 2007 08:09:05PM 1 point [-]

Sounds a lot like this paper by Chris Phoenix.

nothing we discover about quarks is going to throw out Darwinism

Hah, tell that to the people who say "Darwinism" is based on a "debunked deterministic metaphysics".

The "standard degeneration" link is broken.

Comment author: Z._M._Davis 15 December 2007 10:05:43PM 3 points [-]

Caledonian: "Truth does not require guarding."

Doesn't it, though? If a minority that happens to know a truth, but they all keep quiet about it, what's to keep the masses from remaining ignorant indefinitely? Of course (tautologically) the truth will still be true whether or not anyone knows it, but I get the sense that you were implying something less trivial.

Comment author: Caledonian2 15 December 2007 11:05:43PM 0 points [-]

If a minority that happens to know a truth, but they all keep quiet about it, what's to keep the masses from remaining ignorant indefinitely?

In my experience, loudly proclaiming an unfamiliar truth is one of the best ways to keep people ignorant.

The only sorts of knowledge that need to be protected and preserved are the ones that involve contingency: biodiversity, history, paleontology and archeology, etc. can all be harmed if their data is lost. Undying truths can always be rediscovered no matter how many times they're lost.

Comment author: James_Bach 15 December 2007 11:17:14PM 3 points [-]

When you speak of "guardians of truth" I hear "guardians of social order." I don't think the Inquisition thought of truth in epistemic terms, the way we do. They thought of "truth" as the order of the world that was under constant assault by dark forces.

Truth guardianship in science might be understood as defending Kuhnian "normal science" from assault by people outside of the dominant paradigm; or perhaps the process of indoctrinating new scientists in the accepted norms of that paradigm.

Comment author: manuelg 16 December 2007 01:43:56AM 0 points [-]

> The perfect age of the past, according to our best anthropological evidence, never existed.

Minor point: in defense of the esteemed Taoist, I would argue Chuang Tzu was speaking of the time humans were small groups of hunter-gatherers. Based on my understanding of Jared Diamond's "Agriculture: the worst mistake in the history of the human race".

Back on the point of your post. I am not ashamed to say I listen to Zig Ziglar tapes (I probably should be). His folksy way of putting it is "Do you want to be a learner, or learned?" With "learned" implying that you have mastered a system of thought perfectly suited for a receding past.

Comment author: DanielLC 26 November 2011 04:05:21AM 7 points [-]

I would argue Chuang Tzu was speaking of the time humans were small groups of hunter-gatherers.

Small groups of hunter-gatherers were only nice to each other within the group. I would much rather live in a world where it's accepted to lie to your neighbor than one where it's accepted to murder someone who isn't.

Comment author: Kenny 20 January 2013 11:02:58PM 1 point [-]

Hunter-gatherers also resorted to murder in-group too.

Comment author: smijer 10 June 2012 12:16:10PM 1 point [-]

Did Chuang Tzu know that much about the ancient history of humans, really?

Comment author: RPMcMurphy 25 January 2015 02:04:40PM *  0 points [-]


Comment author: JoshuaZ 25 January 2015 04:20:55PM *  0 points [-]

What do you mean by that?

Comment author: P_Decker 16 December 2007 01:59:23AM -2 points [-]

Quite a discussion. There's much that was strung together that might better be sifted into piles.

Moses, for example. Moses was an educated man in charge of a host of ignorant, illiterate peasants who firmly believed in magical thinking. Golden idols, etc. So he knew he couldn't make any headway with them by saying, "I'm an educated gent, and I can tell you you're wrong." He couldn't say, "We're in the desert, you fool, and if you eat that pork that's been lying around for three days, you're gonna get trichinosis." Nope. He said, "GOD SAYS pork is UNCLEAN!" And then it's on to commandments given on the mountain, etc. But the jews were "people of the book," which was their salvation. They valued study of the Torah. Which translated to the value of study in general. And that--and prohibitions against their indulgence in other professions, plus Vatican usury laws--sent them into banking and scholarship and physics and other intellectual pursuits of value to all of us.

Now, as to truth. Most embrace a specific "truth" because it suits their disposition while giving their poor little egos some reason besides accomplishment to feel themselves superior to their neighbors. (Read your Hoffer. And, of course, your Thoreau: "If anything trouble a man, if he has a pain in his bowels even, he forthwith begins reforming the world.") The real question is why one can't be happy in his beliefs unless he can impose them on others. Why, for example--a simple one, admittedly--does someone claim to be allergic to perfumes. Of those who have been tested, the only ones to respond did so only at the threshold of smell. Very strange. Could the real issues be "victimization" and control.

But, of course, the worst of all truths is the current one. Beware, my friends, of the zeitgeist. The temper of times is no guide to rational thought.

Comment author: Rixie 01 November 2012 09:31:36PM *  -1 points [-]

Why would he say that pork was unclean, and not other meats too? Why not say that if you eat old food, you'll get sick? These people aren't stupid. In fact, all of these people lived in the desert. There would not be any of them left if they ate old meat on a regular bases. And why would you assume that they were ignorant and illiterate? Your idea doesn't make sense. And making sense is the whole point of this website.

Are you saying that people defend the truth not because they think it's true but because they're mean, desperate people who want people to believe in them? So, if you defend the truth you're an enemy, don't listen to him, he's mean?

And my eyes water when they're too close to perfumes. You just dissed everyone in the world who's sensitive to strong smells. Not nice, buddy.

Comment author: Michael_Krahn 16 December 2007 02:01:05AM -2 points [-]


I just put up a series of posts about Merton that I think you’d enjoy at:


Comment author: Ian_C. 16 December 2007 03:49:18AM 7 points [-]

If you believe in G-d then you believe in a being that can change reality just by willing it. So therefore you believe it's possible for consciousness to change/control existence.

So that could explain why Guardians fear too many non-believers: they feel threatened by what they perceive as the power of other people's consciousness. They fear that if there are too many non-believers that it might change the truth somehow.

But scientists (Seekers) know that reality is what it is regardless of what other people think, so they don't ascribe so much power to their fellow beings, and therefore don't feel as threatened by them.

Comment author: Rixie 01 November 2012 09:43:34PM -1 points [-]

Christians believe that God doesn't change reality just by willing it. No one really knows how he supposedly created the universe. The theory is that Jesus doesn't perform miracles by bending the laws of nature. I'll explain: Potassium + water = big explosion. But if you added something to the water or to the potassium, you could keep it from exploding. So, on earth, nothing ever happens to water that will turn it into wine. But if God exerted a supernatual force on it, it would, without bending the laws of nature. The idea is that the laws of nature incorporate supernatual meddling, but these are laws that we may never discover because we can't meddle with things supernatualy. God doesn't change reality. And anyways, the Guardians thought that God was all-powerful, and that humans weren't, so I'm not even sure they're thoughts went down that road.

Comment author: IlyaShpitser 01 November 2012 10:21:22PM *  0 points [-]

Presumably God, if He exists, implements this by having a Universe that's inherently stochastic. :)

Comment author: Rixie 01 November 2012 11:43:41PM -1 points [-]

If stochastic means what I think it means (random) then yup! Water + divine intervention = wine. But it had to be weird to attract people's attention. Or there's something about the universe that would make sense of all this that we don't know yet. O_o

Comment author: thrawnca 27 July 2016 01:45:53AM -1 points [-]

If you believe in G-d then you believe in a being that can change reality just by willing it

OK, so by that definition...if you instead believe in a perfect rationalist that has achieved immortality, lived longer than we can meaningfully express, and now operates technology that is sufficiently advanced to be indistinguishable from magic, including being involved in the formation of planets, then - what label should you use instead of 'G-d'?

Comment author: hairyfigment 27 July 2016 05:30:18AM 3 points [-]

Khepri Prime, if the sequel to "Worm" goes the way I hope. More seriously, I don't believe any of that, and physics sadly appears to make some of it impossible even in the far future. Most of us would balk at that first word, "perfect," citing logical impossibility results and their relation to idealized induction. So your question makes you seem - let us say disconnected from the discussion. Would you happen to be assuming we reject theism because we see it as low status, and not because there aren't any gods?

Comment author: entirelyuseless 27 July 2016 02:56:19PM -2 points [-]

"Would you happen to be assuming we reject theism..."

Some LWers reject theism because they see it as low status, some for better reasons, and some do not reject it.

I do have an opinion on your personal motivations as opposed to those of other LWers, but it would be obviously unproductive to give it. So it is also an unproductive question.

Comment author: Jiro 28 July 2016 03:44:49PM 0 points [-]

I'd probably have to invent a name for it. Or I might use the term "godlike being", implying that the being has some, but not all, characteristics in common with what people think of as God.

Comment author: Lumifer 28 July 2016 08:34:50PM 0 points [-]

There's "demigod" or if you like the Eastern flavour, "bodhisattva".

Comment author: Z._M._Davis 16 December 2007 04:53:51AM 1 point [-]

"Undying truths can always be rediscovered no matter how many times they're lost."

At potentially prohibitive expense. Can you imagine trying to start physics over again, from the beginning?

Comment author: Caledonian2 16 December 2007 05:49:48AM 0 points [-]

The greatest expense would be time. Trying to rediscover a science, from the beginning, in one or a handful of generations would require crushing amounts of money and work. If any science is ever so thoroughly lost, it would be hopeless to expect it to be retrieved so rapidly - it could only be a long-term project.

I don't think it would be *too* bad, though. Look at how rapidly science has developed since the end of the Dark Ages.

Comment author: Eliezer_Yudkowsky 16 December 2007 06:01:56AM 4 points [-]

If we instantly forgot everything known about gravity, and even forgot calculus, but we kept the existing economic infrastructure of science and math - computers, professional mathematicians of roughly equivalent skill level who happen not to know calculus, billions of dollars a year in R&D funding, etc. - then calculus would be discovered in a month (though it wouldn't be rigorous), Newton's laws a week later, and Einstein's general relativity before the year was out. That's seriously my best estimate for how long it would take.

Comment author: Aaron_Brown 16 December 2007 06:59:35AM 1 point [-]

I once read an argument (can't find source) that a key component of a zeitgeist is whether it locates its ideals in its future or its past.

David Brin has written about this in his discussions of Star Wars (here and here) and Lord of the Rings.

Comment author: TGGP4 16 December 2007 07:32:08AM 4 points [-]

Beaten to the punch on Brin. Mencius Moldbug provides an interesting contrast on the ideal of "progress", though I can't decide on a specific post to single out.

Regarding the Inquisition, there are some who claim that that our conception of it is the result of the same sort of Protestant "black legend" type propaganda that causes us to think of Gustavus Adolphus as a hero but Wallenstein as a villain. Don't know myself, but I find the idea interesting.

an academic field that stops making active progress tends to turn mean. Which ones were you thinking of? I know you don't want to upset people here, but you seem to be making a claim without providing any evidence. Lysenkoism, perhaps?

The Inquisitors were not Truth-Guardians. They were Doctrine-Guardians. That is a very different matter. Truth does not require guarding. Doctrines often do. I thought that was what made a doctrine a doctrine. At any rate, since the Inquisitors believed it was true, I don't have a problem calling them Truth-Guardians.

Prediction markets are a great way to turn the tables in favor of seekers vs guardians. The guardians will bankrupt themselves trying to keep the "correct" beliefs high in price while the seekers will advantageously extract more and more money from them.

Comment author: Leo_G. 16 December 2007 11:22:06AM 1 point [-]

Eliezer, can we get some confidence intervals on those time estimations? If nothing else, I'd like to know what your thought process is about what would go into rediscovering calculus in a month.

Comment author: Caledonian2 16 December 2007 02:37:39PM -2 points [-]

I'd like to know what his thought processes are on a disaster that could wipe out all hardcopy and memories of a subject, but leave the research infrastructure intact.

Comment author: Pseudonymous2 16 December 2007 10:05:45PM 4 points [-]

"The Inquisition thought they had the truth! Clearly this 'truth' business is dangerous."

The Inquisition was not that unusual. Religious and political loyalties tended to be quite entwined, so most states discriminated against believers in the wrong religion, sometimes banning such religions entirely. This naturally led to people carrying on the old (or new) beliefs in secret.

So the Inquisition was empowered to go looking for those secret heretics.

There were large, bloody and religiously inspired wars in Britain, France and Germany, to name but three.

There were none in Spain or Portugal, so perhaps the Inquisition did more good than you think.

Comment author: Rixie 01 November 2012 11:33:15PM 0 points [-]

This website kinda beats up on Christianity a lot . . . I'm sure that there are plenty of other influental religions to bang on . . .

Comment author: Strange7 14 November 2012 03:42:12AM 1 point [-]

Sure, but Shinto doesn't get so pushy about boneheaded cosmological claims. Mostly they just dance. Where's the fun in arguing with that?

Comment author: Desrtopa 14 November 2012 05:12:25AM 1 point [-]

Well, they do charge for purification rituals and so on (modern Shinto shrines are basically businesses, they have to get their money somewhere,) but I don't think anyone on this site has ever felt pressured to pay for their services.

Comment author: Strange7 15 November 2012 12:10:06AM -1 points [-]

Making a living by performing standardized services for money on the open market is pretty much the opposite of the sort of thought and behavior this site tends to "bang on."

Comment author: Desrtopa 16 November 2012 02:56:33AM 3 points [-]

Well, we've got a lot of libertarian and libertarian-leaning members, but I think a lot of people here are also not so hot on businesses like, say, homeopathy, which provide goods or services on the presumption that they do something that they actually don't.

Comment author: MugaSofer 14 November 2012 09:56:47AM 2 points [-]

How is this "beating up on Christianity"? Pseudonymous is saying that the Inquisition - the main counterargument to the claim that Christianity is good for society - was actually justified. That seems like defending Christianity to me.

Comment author: Laozi 03 June 2008 01:42:55PM -2 points [-]

Chuang Tzu was referring to the hunter-gatherers.

Comment author: AspiringRationalist 05 June 2012 09:03:13PM 0 points [-]

Is there any evidence that he was? How would he have had knowledge of hunter-gatherer tribes? This sounds suspiciously like over-fitting.

Comment author: imd 01 January 2012 09:48:22AM 1 point [-]

Perhaps the difference between the Inquisition and Feynman is that science specifically claims it has nothing to say about morality, so it can't justify killing anyone in its name.

Comment author: PetjaY 24 December 2014 07:26:15PM 0 points [-]

Science has much to say about morality. It can say which morals different groups of people have, what are probable causes for morals, and which morals are useful on an gene|individual|group|society|planet level.

Comment author: Lumifer 24 December 2014 07:54:32PM -1 points [-]

what are probable causes for morals

That's not science, that's story-telling.

which morals are useful on an gene|individual|group|society|planet level

That's not science, that's preaching.

Comment author: TheOtherDave 24 December 2014 09:00:35PM 2 points [-]

On your view, does science have anything to say about the probable causes for gastric enzymes, eyeballs, or toenails?

Comment author: SkyDK 24 March 2012 04:36:23PM 0 points [-]

I think it's easier to go deeper yet simpler. I would say that this is close, but still missing "Questions like that don't have neat single-factor answers. But I would argue that one of the factors has to do with assuming a defensive posture toward the truth, versus a productive posture toward the truth."

and: "When people connect their personal value and self-esteem to a given belief they're prone to persecute anyone claiming that given belief to be false, since they see that claim as undermining their reason to exist".

Or more plainly: Debates become existential struggles for some people, because they ultimately do not separate themselves from their believes and opinions. Hence they react extremely strongly to any perceived threat to their belief or opinion; occasionally as strongly (or stronger) than had their physical person been threatened with extinction.

Comment author: pnrjulius 05 June 2012 04:18:58PM 0 points [-]

I think you have just captured the essence of what makes the Enlightenment culture different from all the others. It's also why people who aren't yet quite sold on the Enlightenment project have so much trouble understanding us; they are used to harkening back to the "good old days", and when we tell them, "No, the past was terrible; you'd die of malaria or get burned at the stake" they don't understand. They think we have no values, because we have no authorities on value. They think we don't believe in truth because we locate the truth in the future instead of the past.

(It doesn't help that there are moral relativists who actually say things like "There is no such thing as truth" and "anyone's values are as good as anyone else's". Maybe we should be spending more time refuting and repudiating such people.)

I even see this among people who mostly accept the basic ideals of rationality and science; they do things like quote Thomas Jefferson as if Jefferson were one of the ancient prophets who knew all the deep truths we have since forgotten. The man owned slaves! He was right about a lot of things, but also wrong about a lot of other things; you should be quoting him only to talk about his ideas, not yours. Similar things happen when people harken back to the US Constitution, or the writings of Ayn Rand. It's not even that wrong---it's surely better than the Bible or the Qur'an---but you're missing the whole point if you hold up a chunk of cellulose and say it's the truth. You should be pointing outside, at the world.

Comment author: christopherj 01 October 2013 05:06:36PM 0 points [-]

I think someone is failing to consider that sometimes truth gets mixed with politics. When truth and politics mix, it suddenly becomes very significant whether other people accept your truth or not. People become not just wrong, but enemy. Can this concept be dis-entwined from the concept of authoritarian guardianship of non-recoverable truth?

For example, I would say that Democrats are supposed to be forward-looking and in favor of knowledge and science, yet I'm quite sure that many Democrats would be opposed to scientific studies that show ineffectiveness of one of their social programs.

Comment author: aphyer 28 May 2014 02:24:19PM 1 point [-]

I forget where I saw this (might actually have been elsewhere on LW?), but I encountered the idea that a component of the backward-reverent ages was the Roman Empire. When your civilization is built on the remnants of Roman roads that are better than anything you can make, it's forgivable to view the world as having fallen from grace.