= 27d567bc2d48d9f40e9ccc20ea370b15)
taw comments on How To Lose 100 Karma In 6 Hours -- What Just Happened - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (214)
Outside view question for anyone with relevant expertise:
It seems to be that lesswrong has some features of early cult (belief that the rest of the world is totally wrong about wide range of subjects, messiah figure, secretive inner circle, mission to save the world etc.). Are ridiculous challenges of group's leadership, met with similarly ridiculous response from it, typical feature of group's gradual transformation into a fully developed cult?
My intuitive guess is yes, but I'm no expert in cults. Anyone has relevant knowledge?
This is outside view question about similar groups, not inside view question about lesswrong itself and why it is/isn't a cult.
In my estimate lesswrong isn't close to the point where such questions would get deleted, but as I said, I'm no expert.
I know a thing or two (expert on Scientology, knowledgeable about lesser nasty memetic infections). In my opinion as someone who knows a thing or two about the subject, LW really isn't in danger or the source of danger. It has plenty of weird bits, which set off people's "this person appears to be suffering a damaging memetic infection" alarms ("has Bob joined a cult?"), but it's really not off on crack.
SIAI, I can't comment on. I'd hope enough people there (preferably every single one) are expressly mindful of Every Cause Wants To Be A Cult and of the dangers of small closed groups with confidential knowledge and the aim to achieve something big pulling members toward the cult attractor.
I was chatting with ciphergoth about this last night, while he worked at chipping away my disinterest in signing up for cryonics. I'm actually excessively cautious about new ideas and extremely conservative about changing my mind. I think I've turned myself into Mad Eye Moody when it comes to infectious memes. (At least in paranoia; I'm not bragging about my defences.) On the other hand, this doesn't feel like it's actually hampered my life. On the other other hand, I would not of course know.
I don't have extensive personal experience with SIAI (spent two weekends at their Visiting Fellows house, attended two meetups there, and talked to plenty of SIAI-affiliated people), but the following have been my impressions:
People there are generally expected to have read most of the Sequences... which could be a point for cultishness in some sense, but at least they've all read the Death Spirals & Cult Attractor sequence. :P
There's a whole lot of disagreement there. They don't consider that a good thing, of course, but any attempts to resolve disagreement are done by debating, looking at evidence, etc., not by adjusting toward any kind of "party line". I don't know of any beliefs that people there are required or expected to profess (other than basic things like taking seriously the ideas of technological singularity, existential risk, FAI, etc., not because it's an official dogma, but just because if someone doesn't take those seriously it just raises the question of why they're interested in SIAI in the first place).
On one occasion, there were some notes on a whiteboard comparing and contrasting Singularitarians and Marxists. Similarities included "[expectation/goal of] big future happy event", "Jews", "atheists", "smart folks". Differences included "popularly popular vs. popularly unpopular". (I'm not sure which was supposed to be the more popular one.) And there was a bit noting that both groups are at risk of fully general counterarguments — Marxists dismissed arguments they didn't like by calling their advocates "counterrevolutionary", and LW-type Singularitarians could do the same with categorical dismissals such as "irrational", "hasn't overcome their biases", etc. Note that I haven't actually observed SIAI people doing that, so I just read that as a precaution.
(And I don't know who wrote that, or what the context was, so take that as you will; but I don't think it's anything that was supposed to be a secret, because (IIRC) it was still up during one of the meetups, and even if I'm mistaken about that, people come and go pretty freely.)
People are pretty critical of Eliezer. Of course, most people there have a great deal of respect and admiration for him, and to some degree, the criticism (which is usually on relatively minor things) is probably partly because people there are making a conscious effort to keep in mind that he's not automatically right, and to keep themselves in "evaluate arguments individually" mode rather than "agree with everything" mode. (See also this comment.)
So yeah, my overall impression is that people there are very mindful that they're near the cult attractor, and intentionally and successfully act so as to resist that.
Sounds like it more so than any other small group I know of!
I would be surprised if less wrong itself ever developed fully into a cult. I'm not so sure about SIAI, but I guess it will probably just collapse at some point. LW doesn't look like a cult now. But what was Scientology like in its earliest stages?
Is there mostly a single way how groups gradually turn into cults, or does it vary a lot?
My intuition was more about Ayn Rand and objectivists than Scientology, but I don't really know much here. Anybody knows what were early objectivists like?
I didn't put much thought into this, it's just some impressions.
Yes, there is. One of the key features of cults is that they make their members sever all social ties to people outside the cult, so that they lose the safeguard of friends and family who can see what's happening and pull them out if necessary. Sci*****ogy was doing that from the very beginning, and Less Wrong has never done anything like that.
Not all, just enough. Weakening their mental ties so they get their social calibration from the small group is the key point. But that's just detail, you've nailed the biggie. Good one.
SIAI staff will have learnt to think in ways that are hard to calibrate against the outside world (singularitarian ideas, home-brewed decision theories). Also, they're working on a project they think is really important. Also, they have information they can't tell everyone (e.g. things they consider decision-theoretic basilisks). So there's a few untoward forces there. As I said, hope they all have their wits about them.
/me makes mental note to reread piles of stuff on Scientology. I wonder who would be a good consulting expert, i.e. more than me.
No, it's much more than that. Scientology makes its members cut off communication with their former friends and families entirely. They also have a ritualized training procedure in which an examiner repeatedly tries to provoke them, and they have to avoid producing a detectable response on an "e-meter" (which measures stress response). After doing this for awhile, they learn to remain calm under the most extreme circumstances and not react. And so when Scientology's leaders abuse them in terrible ways and commit horrible crimes, they continue to remain calm and not react.
Cults tear down members' defenses and smash their moral compasses. Less Wrong does the exact opposite.
What defense against EY does EY strengthen? Because I'm somewhat surprised by the amount I hear Aumann's Agreement Theorem bandied around with regards to what is clearly a mistake on EY's part.
I was talking generally, not about Scientology in particular.
As I noted, Scientology is such a toweringly bad idea that it makes other bad ideas seem relatively benign. There are lots of cultish groups that are nowhere near as bad as Scientology, but that doesn't make them just fine. Beware of this error. (Useful way to avoid it: don't use Scientology as a comparison in your reasoning.)
But that error isn't nearly as bad as accidentally violating containment procedures when handling virulent pathogens, so really, what is there to worry about?
(ducks)
The forbidden topic, obviously.
I'd like to see some solid evidence for or against the claim that typical developing cults make their members cut off communication with their former friends and families entirely.
If the claim is of merely weakening these ties, then this is definitely happening. I especially mean commitment by signing up for cryonics. It will definitely increase mental distance between affected person and their formerly close friends and family, I guess about as much signing up for a weird religion but mostly perceived as benign would. I doubt anyone has much evidence about this demographics?
I don't think they necessarily make them - all that's needed is for the person to loosen the ties in their head, and strengthen them to the group.
An example is terrorist cells, which are small groups with a goal who have gone weird together. They may not cut themselves off from their families, but their bad idea has them enough that their social calibrator goes group-focused. I suspect this is part of why people who decompartmentalise toxic waste go funny. (I haven't worked out precisely how to get from the first to the second.)
There are small Christian churches that also go cultish in the same way. Note that in this case, the religious ideas are apparently mainstream - but there's enough weird stuff in the Blble to justify all manner of strangeness.
At some stage cohesion of the group becomes very important, possibly more important than the supposed point of the group. (I'm not sure how to measure that.)
I need to ask some people about this. Unfortunately, the real experts on cult thinking include several of the people currently going wildly idiotic about cryonics on the Rick Ross boards ... an example of overtraining on a bad experience and seeing a pattern where it isn't.
Regardless of actual chances of both working and considering the issue from purely sociological perspective - signing up for cryonics seems to be to be a lot like "accepting Jesus" / born again / or joining some far-more-religious-than-average subgroups of mainstream religions.
In both situations there's some underlying reasonably mainstream meme soup that is more or less accepted (Christianity / strict mind-brain correspondence) but which most people who accept it compartmentalize away. Then some groups decide not to compartmentalize it but accept consequences of their beliefs. It really doesn't take much more than that.
Disclaimers:
I'm probably in some top 25 posters by karma, but I tend to feel like an outsider here a lot.
The only "rationalist" idea from LW canon I take more or less seriously is the outside view, and the outside view says taking ideas too seriously tends to have horrible consequences most of the time. So I cannot even take outside view too seriously, by outside view - and indeed I have totally violated outside view's conclusions on several occasions, after careful consideration and fully aware of what I'm doing. Maybe I should write about it someday.
In my estimate all FAI / AI foom / nonstandard decision theories stuff is nothing but severe compartmentalization failure.
In my estimate cryonics will probably be feasible in some remote future, but right now costs of cryonics (very rarely honestly stated by proponents, backed by serious economic simulations instead of wishful thinking) are far too high and chances of it working now are far too slim to bother. I wouldn't even take it for free, as it would interfere with me being an organ donor, and that has non-negligible value for me. And even without that personal cost of added weirdness would probably be too high relative to my estimate of it working.
I can imagine alternative universes where cryonics makes sense, and I don't think people who take cryonics seriously are insane, I just think wishful thinking biases them. In non-zero but as far as I can tell very very tiny portion of possible future universes where cryonics turned out to work, well, enjoy your second life.
By the way, is there any reason for me to write articles expanding my points, or not really?
My own situation is not so different although
(a) I have lower karma than you and
(b) There are some LW posters with whom I feel strong affinity
I myself am curious and would read what you had to say with interest and this is a weak indication that others would but of course it's for you to say whether it would be worth the opportunity cost. Probably the community would be more receptive to such pieces if they were cautious & carefully argued than if not; but this takes still more time and effort.
I'm just some random lurker, but I'd be very interested in these articles. I share your view on cryonics and would like to read some more clarification on what you mean by "compartmentalization failure" and some examples of a rejection of the outside view.
Quick reading suggests that Hubbard first founded "dianetics" in late 1949/early 1950, and it became "scientology" only in late 1953/early 1954. As far as I can tell it took them many years until they became Scientology we know. There's some evidence of evaporative cooling at that stage.
And just as David Gerard says, modern Scientology is extreme case. By cult I meant something more like objectivists.
The Wikipedia articles on Scientology are pretty good, by the way. (If I say so myself. I started WikiProject Scientology :-) Mostly started by critics but with lots of input from Scientologists, and the Neutral Point Of View turns out to be a fantastically effective way of writing about the stuff - before Wikipedia, there were CoS sites which were friendly and pleasant but rather glaringly incomplete in important ways, and critics' sites which were highly informative but frequently so bitter as to be all but unreadable.
(Despite the key rule of NPOV - write for your opponent - I doubt the CoS is a fan of WP's Scientology articles. Ah well!)
I don't have a quick comment-length intro to how cults work. Every Cause Wants To Be A Cult will give you some idea.
Humans have a natural tendency to form close-knit ingroups. This can turn into the cult attractor. If the group starts going a bit weird, evaporative cooling makes it weirder. edit: jimrandomh nailed it: it's isolation from outside social calibration that lets a group go weird.
Predatory infectious memes are mostly not constructed, they evolve. Hence the cult attractor.
Scientology was actually constructed - Hubbard had a keen understanding of human psychology (and no moral compass and no concern as to the difference between truth and falsity, but anyway) and stitched it together entirely from existing components. He started with Dianetics and then he bolted more stuff onto it as he went.
But talking about Scientology is actually not helpful for the question you're asking, because Scientology is the Godwin example of bad infectious memes - it's so bad (one of the most damaging, in terms of how long it takes ex-members to recover - I couldn't quickly find the cite) that it makes lesser nasty cults look really quite benign by comparison. It is literally as if your only example of authoritarianism was Hitler or Pol Pot and casual authoritarianism didn't look that damaging at all compared to that.
Ayn Rand's group turned cultish by evaporative cooling. These days, it's in practice more a case of individual sufferers of memetic infection - someone reads Atlas Shrugged and turns into an annoying crank. It's an example of how impossible it is to talk someone out of a memetic infection that turns them into a crank - they have to get themselves out of it.
Is this helpful?