So, I’ve noticed that a significant number of my friends in the Rationalist and Effective Altruist communities seem to stumble into pits of despair, generally when they structure their lives too rigidly around the in-group’s principles. Like, the Rationalists become miserable by trying to govern their entire lives through nothing but rationality, and the EAs feel bad by holding themselves to an impossible standard of ethics. [...]

I’ve tried to figure out why this happens, and I’ve tried to write about it several times, batting around complex notions that, as I examine them, reveal themselves to be fake models that make me sound smart but don’t explain anything.

But today I realized that it’s generally much simpler than I thought previously. Most of it is just toxic social norms. These groups develop toxic social norms. In the Rationalist community, one toxic norm is something like, “you must reject beliefs that you can’t justify, sentiments that don’t seem rational, and woo things.” In the EA community, one toxic norm is something like, “don’t ever indulge in Epicurean style, and never, ever stop thinking about your impact on the world.”

Generally, toxic social norms don’t develop intentionally, nobody wants them to happen, they’re not written down, and nobody enforces them explicitly. (The intentional development of toxic social norms is otherwise known as founding a cult.) What happens is that there are positive social norms, like, “talking about epistemics and being curious about beliefs is cool,” or “try to be intentional about the positive impact you can have on the world.” These norms are great! But then, the group acts like a group, which is to say, people confer status depending on level of apparent adherence to values. This leads insecure people who completely depend on the group to over-identify with the set of values, to the extent that even slightly contrary actions become forbidden. Not forbidden in the like “we’ll arrest you” way, but in the like “everyone in the room immediately looks at you like you’re being rude if you talk about spirituality” way. 

And then the second, more sinister stage occurs—the point at which these toxic norms are internalized such that they apply to you when you’re in a room alone. As Wittgenstein noted, it’s hard to tell where aesthetics end and ethics begin; it can start to feel unethical, like, dirty, to perform behaviors your peers would think distasteful. Toxic norms eventually pervade completely, to the point where you don’t even want to think bad thoughts. 

Sometimes—often—these forbidden thoughts/actions aren’t even contrary to the explicit values. They just don’t fit in with the implied group aesthetic, which is often a much stricter, more menacing guideline, all the more so because it’s a collective unwritten fiction. “Rationality is cool” becomes “rationality is the best framework” becomes “Rationalist and Rationalist-flavored stuff is a better use of your time than anything else” becomes “it’s uncool if you want to spend a lot of time doing stuff that has nothing to do with testable beliefs, or our favorite issues.” This is all unintentional and implicit. No Rationalist has ever said, to my knowledge, that you shouldn’t write poetry, but a few Rationalists have told me that they feel like they shouldn’t make weird art because it’s dumb and un-Rationalist to do so—they feel they ought to produce useful thoughts instead, even though their hearts are trying to steer them somewhere else. I point out to them that Scott Alexander wrote a fantasy novel for fun, but somehow this isn’t persuasive enough.

Here, I should probably stop and define toxic norms. I think a toxic norm is any rule where following it makes you feel like large parts of you are bad. The EA version is thinking that you’re evil if your soul/body/emotions are crying out for you to relax, slack off a bit, and spend money on yourself, because you ought to be spending every possible moment working on human flourishing. I’ve heard tales of people struggling with their decision to buy a tasty snack rather than donate $5 to charity, and, more worryingly, people feeling guilty that they want to have children, since that would distract them from the work of Improving Humanity. This obviously leads to burnout and self-loathing. Meanwhile, the Rationalist version is thinking that you’re stupid and not worth talking to if you yearn for the spiritual/aesthetic/woo/non-justifiable, or if you haven’t been able to come to grips with your issues through rational means. This leads to emotional damage being ignored, intuition being dismissed, and systematizing being preferred inappropriately above all other modes of thinking and feeling.

One sign of toxic social norms is if your behavior does deviate from the standard, you feel that the only way of saving face is through explaining your behavior via the group values. Like, if you watch the Bachelor all the time, and one of your smart peers finds out about that, you might find yourself hastily explaining that the series is enjoyable to you as an applied experiment in evolutionary psychology, when, in fact, you just like social drama because watching humans freak out is fun. I will never forget hearing a Rationalist friend ask a non-Rationalist friend whether he loved riding motorcycles because it was an experiment in social status, rather than, y’know, vroom vroom fun thing go fast.

I’m not mentioning these communities because I think they’re extra toxic or anything, by the way. They’re probably less toxic than the average group, and a lot of their principles are great. Any set of principles, if followed too strictly and assigned too much social value, can become a weird evil super-ego that creeps into the deepest crevices of your psyche. (One must imagine the serene Yogi seized with crippling shame over their perfectly normal road rage.) These groups are just the ones I’m most familiar with right now, and thus the places where I see these patterns most often. In the past, I would’ve used examples from, like, the chess scene, or the artsy prose scene, but I’m not close to those scenes currently, and haven’t been for years, so I’m not even remotely qualified to talk about them. I’ve heard that the startup scene, the internet poker scene, and the crypto scene have all kinds of native pathologies, but someone else will write those essays.

Also, these norms aren’t toxic for everyone! There are a few people who are, in fact, happiest when they’re entirely, or almost entirely, devoted to the fancy intellectual principles of a specialized group. But this is not most people. And this can actually compound the problem! If there are people in the group who are perfect examples of the desired behavior, they can be positive exemplars, but also negative exemplars—constant reminders that you are falling short. (Also, certain group leaders can quietly, and probably unintentionally, inflect the norms in a subtle way, thus accentuating the degree to which they are seen as exemplary, and the degree to which others are seen as inferior.)

This is, perhaps, an inevitable danger for nerdy people. For lots of intellectual weird people that don’t fit in, their first social stage is rejection from society in general, and then, later on, their second social stage is finding understanding in a tightly-knit subculture. And they cling to this subculture like a life-raft and are willing—happy, even—to initially reject any parts of themselves that don’t fit within this new community. And their new peers, unintentionally, facilitate this rejection. They don’t feel that this is toxic, because they feel like they’ve already seen what social toxicity is: it’s the prime normie directive that we learn in school: don’t be weird, ever. [...]

And the people being deferred to—the senior members of the group—don’t want this dynamic at all, but they don’t necessarily notice that it’s happening, because the outward manifestation of this is people being really impressed by you. Like, if you’re big in the EA scene, and a young freshly minted EA can’t stop talking about how excited they are to do good, and how inspired they are by your virtuousness, there’s maybe no obvious sign that they’ve started rejecting every part of themself that is not congruent to this new identity. You would have no reason to worry about that. You would probably just feel good, and glad that your principles are so convincing. So it’s hard to even see this issue sometimes, let alone figure out how to solve it. (Although I’ve heard from Rationalist luminary Aella that some Rationalists are, indeed, beginning to take it seriously, which is great.)

I don’t know whether all of this can be avoided entirely. Part of it is just growing up. It’s regular Kegan Stage 4 stuff. You conceive of who you are by seeing the world through some epistemic/moral lens, usually the one relied upon by the group who abuses you least. Eventually, you notice the flaws in that lens, and then you become your own thing, clearly related to the group, but distinct from it, not easily captured by any label or list of properties. 

(more behind the link)