Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Every Cause Wants To Be A Cult

37 Post author: Eliezer_Yudkowsky 12 December 2007 03:04AM

Followup toCorrespondence Bias, Affective Death Spirals, The Robbers Cave Experiment

Cade Metz at The Register recently alleged that a secret mailing list of Wikipedia's top administrators has become obsessed with banning all critics and possible critics of Wikipedia.  Including banning a productive user when one administrator—solely because of the productivity—became convinced that the user was a spy sent by Wikipedia Review.  And that the top people at Wikipedia closed ranks to defend their own.  (I have not investigated these allegations myself, as yet.  Hat tip to Eugen Leitl.)

Is there some deep moral flaw in seeking to systematize the world's knowledge, which would lead pursuers of that Cause into madness?  Perhaps only people with innately totalitarian tendencies would try to become the world's authority on everything—

Correspondence bias alert!  (Correspondence bias: making inferences about someone's unique disposition from behavior that can be entirely explained by the situation in which it occurs.  When we see someone else kick a vending machine, we think they are "an angry person", but when we kick the vending machine, it's because the bus was late, the train was early and the machine ate our money.)  If the allegations about Wikipedia are true, they're explained by ordinary human nature, not by extraordinary human nature.

The ingroup-outgroup dichotomy is part of ordinary human nature.  So are happy death spirals and spirals of hate.  A Noble Cause doesn't need a deep hidden flaw for its adherents to form a cultish in-group.  It is sufficient that the adherents be human.  Everything else follows naturally, decay by default, like food spoiling in a refrigerator after the electricity goes off.

In the same sense that every thermal differential wants to equalize itself, and every computer program wants to become a collection of ad-hoc patches, every Cause wants to be a cult.  It's a high-entropy state into which the system trends, an attractor in human psychology.  It may have nothing to do with whether the Cause is truly Noble.  You might think that a Good Cause would rub off its goodness on every aspect of the people associated with it—that the Cause's followers would also be less susceptible to status games, ingroup-outgroup bias, affective spirals, leader-gods.  But believing one true idea won't switch off the halo effect.  A noble cause won't make its adherents something other than human.  There are plenty of bad ideas that can do plenty of damage—but that's not necessarily what's going on.

Every group of people with an unusual goal—good, bad, or silly—will trend toward the cult attractor unless they make a constant effort to resist it.  You can keep your house cooler than the outdoors, but you have to run the air conditioner constantly, and as soon as you turn off the electricity—give up the fight against entropy—things will go back to "normal".

On one notable occasion there was a group that went semicultish whose rallying cry was "Rationality!  Reason!  Objective reality!"  (More on this in future posts.)  Labeling the Great Idea "rationality" won't protect you any more than putting up a sign over your house that says "Cold!"  You still have to run the air conditioner—expend the required energy per unit time to reverse the natural slide into cultishness.  Worshipping rationality won't make you sane any more than worshipping gravity enables you to fly.  You can't talk to thermodynamics and you can't pray to probability theory.  You can use it, but not join it as an in-group.

Cultishness is quantitative, not qualitative.  The question is not "Cultish, yes or no?" but "How much cultishness and where?"  Even in Science, which is the archetypal Genuinely Truly Noble Cause, we can readily point to the current frontiers of the war against cult-entropy, where the current battle line creeps forward and back.  Are journals more likely to accept articles with a well-known authorial byline, or from an unknown author from a well-known institution, compared to an unknown author from an unknown institution?  How much belief is due to authority and how much is from the experiment?  Which journals are using blinded reviewers, and how effective is blinded reviewing?

I cite this example, rather than the standard vague accusations of "Scientists aren't open to new ideas", because it shows a battle line—a place where human psychology is being actively driven back, where accumulated cult-entropy is being pumped out.  (Of course this requires emitting some waste heat.)

This post is not a catalog of techniques for actively pumping against cultishness.  Some such techniques I have said before, and some I will say later.  Today I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor.  And that if you can point to current battle lines, it does not mean you confess your Noble Cause unworthy.  You might think that if the question were "Cultish, yes or no?" that you were obliged to answer "No", or else betray your beloved Cause.  But that is like thinking that you should divide engines into "perfectly efficient" and "inefficient", instead of measuring waste.

Contrariwise, if you believe that it was the Inherent Impurity of those Foolish Other Causes that made them go wrong, if you laugh at the folly of "cult victims", if you think that cults are led and populated by mutants, then you will not expend the necessary effort to pump against entropy—to resist being human.

 

Part of the Death Spirals and the Cult Attractor subsequence of How To Actually Change Your Mind

Next post: "Guardians of the Truth"

Previous post: "When None Dare Urge Restraint"

Comments (31)

Sort By: Old
Comment author: Nominull2 12 December 2007 04:00:11AM 4 points [-]

"Resist against being human" is an interesting choice of words. Surely, most people would not see that as a goal worth pursuing.

Comment author: christopherj 07 December 2013 07:11:45AM 7 points [-]

I am annoyed at many things that are part of being human. In no particular order, mortality, enjoying unhealthy foods, having to exercise to be fit, getting scared of things I know to be safe, becoming confrontational when someone is trying to help me with deeply held but wrong beliefs, poor intelligence and especially memory (other people say otherwise), pathetic mathematical abilities (takes longer than the blink of an eye to divide two 100 digit numbers), spending 1/3 of my time asleep, inability to communicate at more than about 0.005 kB/s, bleeding, requiring an environment with a narrow range of acceptable temperatures, atmospheres, and gravity, and I'm pretty sure I could fill up several pages with things I don't like about being human.

Comment author: Caledonian2 12 December 2007 04:19:56AM 7 points [-]

Surely, most people would not see that as a goal worth pursuing.

If you're going to concern yourself with popularity contests, you might as well abandon this entire field of endeavor right now. What "most people see" is utterly irrelevant.

Comment author: hamnox 28 June 2011 06:36:27PM 9 points [-]

And yet it's a true observation, and entirely relevant if you're going to concern yourself with convincing other people to resist against being human.

Comment author: TGGP4 12 December 2007 07:12:31AM 4 points [-]

I wonder if a Randian will pop up to deny any assertions of cultishness.

I liked your bit about science. I get tired of people saying "Science is a religion too" or some variant thereof, whether from Christians or global warming skeptics like Arnold Kling.

Comment author: pnrjulius 20 April 2012 01:45:12PM 0 points [-]

If Randians do pop up, the lady doth protest too much methinks.

Comment author: Unknown 12 December 2007 07:44:12AM 1 point [-]

This is a great post. Especially since it applies just as much to the cause of "overcoming bias."

Comment author: Stuart_Armstrong 12 December 2007 08:30:04AM 7 points [-]

whose rallying cry was "Rationality! Reason! Objective reality!"

Not to disagree with your main point (I've seen cultishness even in mathematics, where we really do have objective reality), but aren't those cults whose banner is Rationality in a better position than those who aren't? They may be just as cultish on the inside, but they have publicly accepted a standard that makes then vulnerable to criticism they cannot just dismiss. Wouldn't that make them a bit more honest?

Same point: are we more honest at overcoming bias, because we have a type of discourse that leaves us vulnerable to arguments of bias in ways we can't ignore - or do we just become more skilled at rationalising?

Comment author: Bluehawk 17 April 2012 12:59:40AM *  1 point [-]

That's a question that everybody here needs to ask themselves every time they post, if they're to fight the good fight against cult-entropy.

Comment author: Paul_Ganssle 12 December 2007 09:12:21AM 4 points [-]

TGGP: Not to defend global warming denialism, but is that the entirety of your evidence that Kling is a denier? Because when I read that, I strongly got the impression that he was not rallying against people who believe that anthropogenic global warming is occuring, but rather against people preaching dogmatic versions of a nuanced science. I didn't get the impression that he was saying that global warming scientists are preaching a religion, but that global warming activists are, and I think that's completely reasonable. I mean, there's a difference between calling out science and calling out activists: One thing to note is that Al Gore has been on the global warming beat since BEFORE there was a scientific consensus about it (at least as far as he tells it). I don't want to go off on Al Gore too much, but that's certainly the sign of a dogmatist (that is, believing something to be true before the world's experts on the subject had come to a consensus about it). There were basically two dogmas on the issue, and if you picked randomly you'd have a 50% chance of being vindicated.

And I hate to have to reiterate this, but I'm afraid to be lumped in with global warming deniers because I am defending someone who is perceived to be one, but I do NOT find the denialist position compelling. I do however think that Kling makes a good point there (and a similar point to the one in this blog post, I might add) that it is important to convey how you know what you know. It might be reading too far into it, but I would say that that circles back to the point made in this post about cultishness: it's easy to say that something "good" like trying to prevent climate disasters isn't going to have those cultish aspects of attempting to suppress dissent and form in-group mentality, and it it is important (if you are interested in overcoming your biases) to work against this by quantifying how big of a cultish presence you have in your "good" cause.

Comment author: Matt_Heath 12 December 2007 12:43:23PM 1 point [-]

The I found the main points of the article interesting and fairly convincing but you seem to over-correct for correspondence bias when you say "If the allegations about Wikipedia are true, they're explained by ordinary human nature, not by extraordinary human nature". Even if normal human behaviour leads to cultishness, why assume that individual psychological quirks didn't have a relevant effect in a specific case?

Comment author: pnrjulius 20 April 2012 01:49:19PM 1 point [-]

It's certainly possible to overcompensate for the fundamental attribution error.

This is what I think happens when people say things like "Stalin was just a product of his circumstances." No, he was a manipulative, sadistic psychopath; his circumstances are what made him a world leader and mass murderer instead of a corrupt banker or serial killer.

But in this case, I do think that the admins of Wikipedia are humans of at least normal---if not in fact above-average---moral character, falling prey to their circumstances. Their behavior does not seem SO extreme, SO cruel, that it can't be fit with what we know about normal human beings.

Comment author: steven 12 December 2007 01:27:36PM 3 points [-]

TGGP, Kling wasn't calling science a religion, he was calling the anti-global-warming movement a religion, which strikes me as true regardless of whether the skeptics are right about the science (I think they're not).

Comment author: RobinHanson 12 December 2007 02:03:40PM 8 points [-]

So can we learn to recognize the sound of a "cult cooler", cooling down the cultishness, and distinguish it from a fake recording of such? Or at least invent a cultometer, so we can check our cultempature?

Comment author: timtyler 19 December 2009 04:36:13AM 4 points [-]
Comment author: themusicgod1 18 December 2013 12:25:32AM *  1 point [-]

Your second link is broken. In addition to the Internet archive I have posted a blog post inspired by some of my experiences with a cult, containing the article in its entirety for posterity.

Comment author: TGGP4 12 December 2007 05:25:29PM 1 point [-]

I singled out Kling because I could remember several occasions where he used the phrase "the religion of global warming" or something like it and just linked to that post after some quick googling. Perhaps my memory is bad and he hasn't used it that many times though.

Comment author: Nick_Tarleton 12 December 2007 07:08:15PM 2 points [-]

In the same sense that every thermal differential wants to equalize itself, and every computer program wants to become a collection of ad-hoc patches, every Cause wants to be a cult. It's a high-entropy state into which the system trends, an attractor in human psychology.

For this to be strictly true, there would have to be more cultish microstates, more possible cultish groups, than sane ones. Do you think that's actually the case?

Aren't those cults whose banner is Rationality in a better position than those who aren't? They may be just as cultish on the inside, but they have publicly accepted a standard that makes then vulnerable to criticism they cannot just dismiss.

They can still dismiss it by redefining "rationality" to exclude the methods the attacker is using.

Comment author: aspera 15 October 2012 07:55:07PM 0 points [-]

That thought occurred to me too, and then I decided that EY was using "entropy" as "the state to which everything naturally tends" But after all, I think it's possible to usefully extend the metaphor.

There is a higher number of possible cultish microstates than non-cultish microstates, because there are fewer logically consistent explanations for a phenomenon than logically inconsistent ones. In each non-cultish group, rational argument and counter-argument should naturally push the group toward one describing observed reality. By contrast, cultish groups can fill up the rest of concept-space.

Comment author: manuelg 12 December 2007 07:08:39PM 5 points [-]

> "Resist against being human" is an interesting choice of words. Surely, most people would not see that as a goal worth pursuing.

Nominull nailed it on the head, Eliezer. What are the human qualities worth amplifying, and what are the human qualities worth suppressing?

For myself, "cultishness" is definitely a human group failure mode.

To others, maybe "cultishness" is a comfortable state of being, like relaxing in a warm bath. (Partake in the vivid imagery of a group of nude people in a drowsy state soaking in their collective body-temperature urea solution...)

I assert that the choice of what elements of humanity are worthy, and what are unworthy, is completely personal and subjective. I would be interested in seeing the argument for the differentiation being objective. Is there an objective criteria for what elements of humanity are worthy, and what are unworthy?

A different point: You really demonstrate the value of blogging and independently developing a stable of ideas, and then being able to reference those ideas with terminology backed up by a hyperlink. I am constantly rereading your posts as you link back to them, and it is interesting and profitable.

Comment author: poke 12 December 2007 07:14:26PM -1 points [-]

We really need to figure out how to create more cultishness. If you could build a cult around known science, which happily describes everything in human experience, and spread it, you'd do more good in the world than "rationality" or "overcoming bias" ever could.

Comment author: sidhe3141 11 November 2010 05:45:53PM 7 points [-]

No. Part of the definition of a cult is an unquestionable dogma, which runs counter to the core ideas of science. Building a cult around known science (even if you understand the principles well enough to avoid engaging in cargo cult science) is going to slow progress.

Comment author: Bluehawk 17 April 2012 01:11:06AM 0 points [-]

Consider replacing "core ideas of science" with "core ideas of society" and I'll wager that's closer to the commonly-used meaning of "cult".

Comment author: dlthomas 17 April 2012 03:56:27AM 2 points [-]

Dropping in mid thread, but I think you parsed that differently than intended; I read it as saying that the notion of unquestionable dogma runs counter to the core ideas of science, not that the dogma itself must run counter to anything in order to be a cult.

Comment author: Bluehawk 25 April 2012 08:39:26PM 1 point [-]

Ah. Yeah, I may have parsed that one incorrectly, now that you mention it. Thanks for pointing that out.

Comment author: josh 12 December 2007 07:19:39PM 1 point [-]

"So can we learn to recognize the sound of a "cult cooler", cooling down the cultishness, and distinguish it from a fake recording of such? Or at least invent a cultometer, so we can check our cultempature?"

We could just ask our perfect Baysian leaders. They know all and understand all.

Comment author: David_J._Balan 12 December 2007 07:38:00PM 4 points [-]

Eliezer's reminder that even rationalists are human, and so are subject to human failings such as turning a community into a cult, is welcome. But it's a big mistake to dismiss explanations such as "Perhaps only people with innately totalitarian tendencies would try to become the world's authority on everything." There is a huge degree of heterogeneity across people in every relevant metric, including a tendency toward totalitarianism. I can't imagine that anyone disputes this. And if the selection process for being in a certain position tends to advantage people with those tendencies, so that they are selected into them, that might well explain a large part of how people in those positions behave.

Comment author: George_Weinberg2 12 December 2007 08:49:24PM 1 point [-]

Or at least invent a cultometer, so we can check our cultempature?

It's a bad sign if we develop identifiable cliques. Because of general attitudes it stands to reason agreements and disagreements won't be randomly distributed, but ideally we shouldn't "agree" or "disagree" with others because we agreed or disagreed with them in the past. It probably wouldn't be too hard to develop some sort of voting software that measured cliquishness if there's a demand for it.

Of course, the real disaster would be if people start saying things like "Eliezer is always right". Nobody is always right.

Comment author: Carl_Shulman 12 December 2007 10:09:28PM 6 points [-]

"Of course, the real disaster would be if people start saying things like "Eliezer is always right". Nobody is always right.

Posted by: George Weinberg | December 12, 2007 at 03:49 PM "

George,

That understates the risk, since self-identified rationalists familiar with the literature would concoct much better rationalizations. For instance, someone might say that "Nick Bostrom is very intelligent, actively works to overcome biases, and seems to have been relatively successful at it. Since almost all top academics are not immersed in the heuristics and biases literature and committed to Overcoming Bias, in a sustained dispute between top academics and Nick Bostrom we should expect the latter to be right much more often than a random high quality academic dissenter," but then treat this as license to accept Bostrom's positions on an improbable number of independent disagreements.

Comment author: Eliezer_Yudkowsky 12 December 2007 10:14:43PM 7 points [-]

And even if Bostrom is always right, repeating back what Bostrom says may not mean that you have acquired any of Bostrom's beliefs. Works great for ingroup identification though.

Comment author: pnrjulius 20 April 2012 01:44:44PM 1 point [-]

I strongly endorse this post. I've actually watched it happen: Groups dedicated to secularism or the Singularity or even rationality itself can degrade into cliques, evolve into tribes, and then ultimately become as much cults as their greatest foes.