You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

private_messaging comments on Notes on Brainwashing & 'Cults' - Less Wrong Discussion

35 Post author: gwern 13 September 2013 08:49PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (101)

You are viewing a single comment's thread. Show more comments above.

Comment author: private_messaging 14 September 2013 04:44:19PM *  2 points [-]

The low rate of retention is extreme filtering. The cults try to get members to sever ties with the family and friends, for example - and this is a filter, most people get creeped out and a few go through with it. edit: and of course, with such extreme filtering, one needs a lot of proselytism to draw just a hundred very dedicated supporters.

Comment author: gwern 14 September 2013 06:35:55PM 4 points [-]

The low rate of retention is extreme filtering.

You are arguing by definition here; please consider what could falsify your mental model of cults. If my local gym discovers only 1% of the people joining after New Years will stick around for more than a year, does that necessarily imply that the gym is ruled by a charismatic leader driving people away so as to maximize the proportion of unthinkingly loyal subordinates?

Low rate of retention is simply low rate of retention. This can be for a great many reasons, such as persecution, more attractive rival organizations, members solving their problems and leaving, or (way down the list) extreme filtering for loyalty which drives away otherwise acceptable members. How often do you see a cult leader going 'well, sure, we could have thousands more members if we wanted (people are pounding down the doors to convert), and majorly increase our donations and financial holdings, but gosh, we wouldn't want to sell out like that!'

Of course, like any organization, there's concerns about freeriding and wasting club goods and it'll seek to strike a balance between inclusiveness and parasite load; but a cult which has 'successfully' shed all but a few fanatics is a cult which is about to become history.

The cults try to get members to sever ties with the family and friends, for example

Recruiting through family and friends is a major strategy of cults - indeed, perhaps the only strategy which does not have abysmally low success rates.

Comment author: private_messaging 14 September 2013 06:59:26PM 1 point [-]

Low rate of retention is a product of many reasons simultaneously, including the extreme weird stuff creeping people out. If your local gym is creepy, it will have lower retention rate, than same gym that is not creepy.

My mental model of failed retention includes the general low retention rate, in combination with the weird things that cult does creeping people out, on top of that.

How often do you see a cult leader going 'well, sure, we could have thousands more members if we wanted (people are pounding down the doors to convert), and majorly increase our donations and financial holdings, but gosh, we wouldn't want to sell out like that!'

I rarely see people reflect on their motives or goal structure. You often see a cult leader abusing a cultist, which leads insufficiently dedicated cultists to leave. Such actions sacrifice quantity for "quality".

Recruiting through family and friends is a major strategy of cults - indeed, perhaps the only strategy which does not have abysmally low success rates.

Yes, and a lot of the time that fails, and the family members start actively denouncing the cult, and the member has to choose between the family and friends, and the cult, at which point, well, few choose the cult.

Comment author: gwern 14 September 2013 10:31:36PM 2 points [-]

Low rate of retention is a product of many reasons simultaneously, including the extreme weird stuff creeping people out.

As pointed out in the OP by one author, the cults in question have in many ways been assimilated by the mainstream and so are far less 'weird' than ever before. Has that helped their retention rates? Environmentalism and meditation are completely mainstream now, have the Hare Krishnas staged a comeback?

If your local gym is creepy, it will have lower retention rate, than same gym that is not creepy.

The counterfactual is not available or producible, and so this is meaningless to point out. If the Hare Krishnas did not hold 'creepy' beliefs, in what sense is this counterfactual organization similar to the Hare Krishnas? If Transcendental Meditators did not do as weird a thing as meditate, how are they Transcendental Meditators? Defining away all the unique characteristics does not add any insight.

You often see a cult leader abusing a cultist, which leads insufficiently dedicated cultists to leave.

"You often see a boss abusing a subordinate, which leads insufficiently dedicated employees to leave. This is because bosses wish to sacrifice quantity and being able to handle work for 'quality' of subordinates."

No, there is nothing unique about cults in this respect. Monkeys gonna monkey. And for the exact same reason businesses do not casually seek to alienate 99% of their employees in order to retain a fanatical 1%, you don't see cults systematically organization-wide try to alienate everyone. You see a few people in close proximity to elites being abused. Just like countless other organizations.

the member has to choose between the family and friends, and the cult, at which point, well, few choose the cult.

Which explains the success of deprogrammers, amirite?

Comment author: private_messaging 14 September 2013 11:25:17PM *  -1 points [-]

I don't see how environmentalism or for that matter meditation itself is creepy.

What's creepy about Hare Krishnas is the zoned out sleep deprived look on the faces (edit: I am speaking of the local ones, from experience), and the whole obsession with the writings of the leader thing, and weirdly specific rituals. Now that environmentalism and meditation are fairly mainstream, you don't have to put up with the creepy stuff if you want to be around people who share your interests in environmentalism and meditation. You have less creepy alternatives. You can go to a local Yoga class, that manages to have same number of people attending as the local Khrishna hangout, despite not trying nearly as hard to find new recruits. You can join a normal environmentalist group.

No, there is nothing unique about cults in this respect. Monkeys gonna monkey. And for the exact same reason businesses do not casually seek to alienate 99% of their employees in order to retain a fanatical 1%, you don't see cults systematically organization-wide try to alienate everyone. You see a few people in close proximity to elites being abused. Just like countless other organizations.

The difference is, of course, in extent. For example, putting up a portrait of the founder at every workplace (or perhaps in a handbook, or the like) would be something that a cult leader would do in a cult, but what a corporation would seldom ever do because doing so would be counter-productive.

edit: actually. What do you think makes joining a cult worse than joining a club, getting a job, and so on? Now, what ever that is, it makes it harder to get new recruits, and requires more dedication.

Comment author: gwern 15 September 2013 06:35:22PM *  3 points [-]

I don't see how environmentalism or for that matter meditation itself is creepy.

Which goes to show how far into the zeitgeist they've penetrated. Go back to the 1960s when the cult panic and popular image of cults was being set, and things were quite different. One of the papers discusses a major lawsuit accusing the Hare Krishnas of 'brainwashing' a teen girl when she ran away from home and stayed with some Krishnas; the precipitating event was her parents getting angry about her meditating in front of a little shrine, and ripping it out and burning it (and then chaining her to the toilet for a while). To people back then, 'tune in, turn on, drop out' sounds less like a life choice than a threat...

What's creepy about Hare Krishnas is the zoned out sleep deprived look on the faces (edit: I am speaking of the local ones, from experience)

Well, I can hardly argue against your anecdotal experiences.

the whole obsession with the writings of the leader thing,

Supreme Court - jurists or cultists? Film at 11. We report, you decide.

and weirdly specific rituals.

I don't even know what 'weirdly specific' would mean. Rituals are generally followed in precise detail, right down to the exact repetitive wording and special garments like Mormon underpants; that's pretty much what distinguishes rituals from normal activities. Accepting Eucharist at mass? Ritual. Filling out a form at the DMV? Not ritual.

You can go to a local Yoga class, that manages to have same number of people attending as the local Khrishna hangout, despite not trying nearly as hard to find new recruits.

Hmm, where was one to find yoga back then... Ah yes, also in cults. Ashrams in particular did a lot of yoga. Interesting that you no longer have to go to an ashram or fly to India if you want to do yoga. It's almost like... these cult activities have been somehow normalized or assimilated into the mainstream...

You can join a normal environmentalist group.

And where did these environmentalist groups come from?

For example, putting up a portrait of the founder at every workplace (or perhaps in a handbook, or the like) would be something that a cult leader would do in a cult, but what a corporation would seldom ever do because doing so would be counter-productive.

Really? That seems incredibly common. Aside from the obvious examples of many (all?) government offices like post offices including portraits of their supreme leader - I mean, President - you can also go into places like Walmart and see the manager's portrait up on the wall.

What do you think makes joining a cult worse than joining a club, getting a job, and so on?

Personally? I think it's mostly competition from the bigger cults. Just like it's hard to start up a business or nonprofit.

Comment author: private_messaging 15 September 2013 06:45:08PM *  -2 points [-]

What do you think makes joining a cult worse than joining a club, getting a job, and so on?

Personally? I think it's mostly competition from the bigger cults. Just like it's hard to start up a business or nonprofit.

That doesn't even make sense as an answer. Rest likewise doesn't seem in any way contradictory to the point I am making, but is posed as such.

Comment author: gwern 15 September 2013 06:47:41PM *  3 points [-]

That doesn't even make sense as an answer.

Of course it makes sense. As I've already claimed, cults are not engaged in some sort of predatory 'brainwashing' where they exploit cognitive flaws to just moneypump people with their ultra-advanced psychological techniques: they offer value in return for value received, just like businesses need to offer value to their customers, and nonprofits need to offer some sort of value to their funders. And these cults have plenty of established competition, so it makes sense that they'd usually fail. Just like businesses and nonprofits have huge mortality rates.

Rest likewise doesn't seem in any way contradictory to the point I am making, but is posed as such.

I've given counter-examples and criticized your claims. Seems contradictory to me.

Comment author: private_messaging 15 September 2013 07:04:37PM *  -2 points [-]

Of course it makes sense.

The question was, "What do you think makes joining a cult worse than joining a club, getting a job, and so on?" . How is competition from other cults impacting the decision to join a cult - any cult?

As I've already claimed, cults are not engaged in some sort of predatory 'brainwashing' where they exploit cognitive flaws to just moneypump people with their ultra-advanced psychological techniques: they offer value in return for value received

Well, I know of one cult that provides value in form of the nice fuzzy feeling of being able - through a very little effort - to see various things that, say, top physicists can not see. Except this feeling is attained entirely through self deception, unbeknown to the individuals, and arguing that it is providing value is akin to arguing that a scam which sells fake gold for the cheap is providing value.

(Then there's of course Janestown, and so on and so forth)

Comment author: gwern 15 September 2013 07:14:03PM *  4 points [-]

How is competition from other cults impacting the decision to join a cult?

Exactly as I said, pressure from other cults: direct retaliation (like the legal system endorsing your kidnapping), opportunity costs, lack of subsidies, regulatory capture being used against you, the risk of joining a small new organization... Many of the reasons that apply to not joining a startup and instead working at Microsoft can be tweaked to apply to small cults vs big cults.

Well, I know of one cult that provides value in form of the nice fuzzy feeling of being able - through a very little effort - to see various things that, say, top physicists can not see. Except this feeling is attained entirely through self deception, unbeknown to the individuals, and arguing that it is providing value is akin to arguing that a scam which sells fake gold for the cheap is providing value.

You know what's even more awesome than self-deception? Sliming people you don't like as cults, when your ideas about what a cult is aren't even right in the first place. Sweet delicious meta-contrarianism.

True, it's not as good a racket as Singer getting paid tons of money to testify about how awful cults are and how powerful their deceptions are - but it's a lot less work and more convenient.

Comment author: Luke_A_Somers 16 September 2013 03:39:30PM 0 points [-]

I wasn't around in the 60s and wasn't aware for any of the 70s, but... Environmentalism seems qualitatively different from everything else here. Is there some baggage to this beyond, say, conservation, or assigning plants and animals some moral weight, that is intended here?

Something may have seemed weirder in the past because it was weirder back then.

I suspect few modern Christians would sign up for AD 200 Christianity.

Comment author: gwern 16 September 2013 04:33:59PM 3 points [-]

Environmentalism seems qualitatively different from everything else here. Is there some baggage to this beyond, say, conservation, or assigning plants and animals some moral weight, that is intended here?

Not really, aside from the standard observation that you can just as easily play the 'find cult markers' game with environmental groups like Greenpeace or ELF. Cleansing rituals like recycling, intense devotion to charismatic leaders, studies of founding texts like Silent Spring, self-abnegating life choices, donating funds to the movement, sacralization of unusual objects like owls or bugs, food taboos ('GMOs'), and so on and so forth.

Comment author: Jiro 15 September 2013 05:05:34PM -1 points [-]

Environmentalism and meditation are completely mainstream now, have the Hare Krishnas staged a comeback?

I would suggest that if beliefs believed by cults becoime mainstream, that certainly decreases one barrier to such a cult's expansion, but because there are additional factors (such as creepiness) that alone is not enough to lead the cult to expand much. It may be that people's resistance to joining a group drastically increases if the group fails any one of several criteria. Just decrementing the number of criteria that the group fails isn't going to be enough, if even one such criterion is left.

"You often see a boss abusing a subordinate, which leads insufficiently dedicated employees to leave. This is because bosses wish to sacrifice quantity and being able to handle work for 'quality' of subordinates.

The level of abuse done by bosses and cult leaders is different, so although the statement is literally true for both bosses and cult leaders, it really doesn't imply that the two situations are similar.

Comment author: gwern 15 September 2013 06:45:06PM 1 point [-]

It may be that people's resistance to joining a group drastically increases if the group fails any one of several criteria.

Maybe, but I don't know how we'd know the difference.

The level of abuse done by bosses and cult leaders is different, so although the statement is literally true for both bosses and cult leaders, it really doesn't imply that the two situations are similar.

Is it really? Remember how many thousands of NRMs there are over the decades, and how people tend to discuss repeatedly a few salient examples like Scientology. Can we really compare that favorably regular bosses with religious figures? Aside from the Catholic Church scandal (with its counterparts among other closemouthed groups like Jewish and Amish communities), we see plenty of sexual scandals in other places like the military (the Tailhook scandal as the classic example, but there's plenty of recent statistics on sexual assault in the military, often enabled by the hierarchy).

Comment author: ChristianKl 14 September 2013 08:26:46PM 2 points [-]

The cults try to get members to sever ties with the family and friends, for example - and this is a filter, most people get creeped out and a few go through with it.

I'm not sure whether that's true. You have people on LessWrong talking about cutting family ties with nonrational family members and nobody get's creeped out.

I don't think I have ever witnessed people getting creeped out by such discussions in the self help area and I think I have frequently heard people encouraging others to cut ties with someone that "holds them back".

Comment author: yli 15 September 2013 01:42:40AM *  5 points [-]

Really? Links? A lot of stuff here is a bit too culty for my tastes, or just embarassing, but "cutting family ties with nonrational family members"?? I haven't been following LW closely for a while now so I may have missed it, but that doesn't sound accurate.

Comment author: Douglas_Knight 15 September 2013 03:38:52AM 3 points [-]

Here's an example.

Comment author: Mestroyer 15 September 2013 02:56:56PM 3 points [-]

diegocaleiro didn't just say they were just irrational:

(1) Stupid (2) Religious (3) Non-rationalists (4) Absolutely clueless about reality (5) Pushy about inserting their ideas/ideals/weltenshaaung/motifs into you?

I strongly suspect that this isn't a case of "My family members don't believe as I do, therefore fuck those guys." but rather "These family members know that I am nonreligious and aggressively proselytize because of it." This probably isn't even about rationality or LessWrong, rather atheism.

Note also that it is diegocaleiro who initiated the conversation, and note the level of enthusiasm about the idea received from other posters (Only ChristianKI and Benito's responses seem wholly in favor, VilliamBur and drethelin's responses are against, shminux and BenLandauTaylor's responses are neutral).

Comment author: Eugine_Nier 15 September 2013 03:37:49PM 4 points [-]

"These family members know that I am nonreligious and aggressively proselytize because of it."

Outside view: These family members know that [diegocaleiro joined a group with weird non-mainstream religious beliefs] and [are trying to deconvert him].

Comment author: yli 15 September 2013 04:01:21AM *  2 points [-]

Thanks for the link. I don't really see creepy cult isolation in that discussion, and I think most people wouldn't, but that's just my intuitive judgment.

Comment author: ChristianKl 15 September 2013 09:56:07AM 6 points [-]

That's the point. It doesn't look that way from the inside.

If someone would tell those family members that the OP cutted their family ties with them because he made a rational analysis with help from his LessWrong friends those family member might see it as an example of the evil influence that LessWrong has on people.

Comment author: Costanza 14 September 2013 08:48:23PM 6 points [-]

I'm at least mildly creeped out by occasional cultish behavior on LessWrong. But every cause wants to be a cult

Eliezer said so, so therefore it is Truth.

Comment author: wedrifid 15 September 2013 02:10:17AM *  -1 points [-]

I'm not sure whether that's true. You have people on LessWrong talking about cutting family ties with nonrational family members and nobody get's creeped out.

I do not believe you. If it is the case that people talk about cutting family ties with 'nonrational family members' then there will be people creeped out by it.

Note that if the 'nonrational' family members also happen to be emotionally abusive family members this would not match the criteria as I interpret it. (Even then I expect some people to be creeped out by the ties cutting and would expect myself to aggressively oppose such expressions so as to suppress a toxic influence.)

Comment author: Eugine_Nier 15 September 2013 12:55:11PM 5 points [-]

Note that if the 'nonrational' family members also happen to be emotionally abusive family members this would not match the criteria as I interpret it.

You do realize that a lot of cults tend to classify normal family reactions, e.g., attempting to get the person out of the cult, as emotional abuse.

Comment author: wedrifid 15 September 2013 01:33:36PM *  3 points [-]

You do realize that a lot of cults tend to classify normal family reactions, e.g., attempting to get the person out of the cult, as emotional abuse.

I don't care and I'm somewhat outraged at this distortion of reasoning. It is so obviously bad and yet remains common and is all too seldom refuted. Emotional abuse is a sufficiently well defined thing. It is an undesirable thing. Various strategies for dealing with it are possible. In severe cases and in relationships where the gains do not offset the damage then severing ties is an appropriate strategy to consider. This doesn't stop being the case if someone else also misuses the phrase 'emotional abuse'.

Enduring emotional abuse rather than severing ties with the abuser because sometimes cultists sever ties while using that phrase is idiotic. Calling people 'creepy' for advocating sane, mainstream interpersonal strategies is absurd and evil.

Comment author: Kaj_Sotala 18 September 2013 05:44:00AM *  4 points [-]

I don't care and I'm somewhat outraged at this distortion of reasoning. It is so obviously bad and yet remains common and is all too seldom refuted.

Sorry, exactly what is it that you're outraged about? Eugene seemed to merely be pointing out that people inside particular social groups might see things differently than people outside them, with the outsiders being creeped out and insiders not being that. More specifically, that things that we deem okay might come off as creepy to outsiders. That seems correct to me.

Comment author: wedrifid 18 September 2013 06:44:59AM 0 points [-]

Sorry, exactly what is it that you're outraged about?

As a general policy:

  • All cases where non-sequitur but technically true claims are made where the actual implied rhetorical meaning is fallacious. Human social instincts are such that most otherwise intelligent humans seem to be particularly vulnerable to this form of persuasion.
  • All arguments or insinuations of the form "Hitler, Osama Bin Laden and/or cultists do <something superficially similar to X>. Therefore, if you say that <X> is ok then you are Bad."
  • Additional outrage, disdain or contempt applies when:
    • The non-sequitur's are, through either high social skill or (as in this case) plain luck, well calibrated to persuade the audience despite being bullshit.
    • Actual negative consequences can be expected to result from the epistemic damage perpetrated.
Comment author: Kaj_Sotala 20 September 2013 11:38:45AM *  5 points [-]

Thanks, that sounds reasonable. I didn't interpret Eugene's comments as being guilty of any of those, though.

Comment author: Eugine_Nier 19 September 2013 07:28:43AM -1 points [-]

All cases where non-sequitur but technically true claims are made where the actual implied rhetorical meaning is fallacious. Human social instincts are such that most otherwise intelligent humans seem to be particularly vulnerable to this form of persuasion.

In my experience nearly all accusations that someone is being "emotionally abusive" are of this type.

Comment author: wedrifid 19 September 2013 10:52:03AM *  2 points [-]

In my experience nearly all accusations that someone is being "emotionally abusive" are of this type.

If that is true then you are fortunate to have lived such a sheltered existence. If it is not true (and to some extent even if it is) then I expect being exposed to this kind of denial and accusation of dishonesty to be rather damaging to those who are actual victims of the phenonemon you claim is 'nearly all' fallacious accusation.

Comment author: Eugine_Nier 19 September 2013 08:31:58PM -1 points [-]

If that is true then you are fortunate to have lived such a sheltered existence.

I could say the same thing about you if you've never encountered people willing to make false accusations of abuse (frequently on behalf of children) with the force of the law, or at least child services behind them.

If it is not true (and to some extent even if it is) then I expect being exposed to this kind of denial and accusation of dishonesty to be rather damaging to those who are actual victims of the phenonemon you claim is 'nearly all' fallacious accusation.

This is as good a summery of the "how dare you urge restraint" position as any I've heard.

Comment author: Eugine_Nier 15 September 2013 03:29:46PM 6 points [-]

Emotional abuse is a sufficiently well defined thing. It is an undesirable thing.

So could you provide a definition. The article you linked to begins by saying:

As of 1996, There were "no consensus views about the definition of emotional abuse."

And then proceeds to list three categories that are sufficiently vague to include a lot of legitimate behavior.

Enduring emotional abuse rather than severing ties with the abuser because sometimes cultists sever ties while using that phrase is idiotic.

You don't seem to be getting the concept of "outside view". Think about it this way: as the example of cults shows, humans have a bias that makes them interpret Bob attempting to persuade Alice away from one's meme set as emotional abuse. Consider the possibility that you're also suffering from this bias.