Comment Permalink

Valentine1y8-1

…I think another pretty good option is "a master rationalist would definitely avoid surrounding themselves with con artists and frauds and other adversarial actors".

I think that's a great option. I'd question a "master rationalist's" skills if they couldn't avoid such adversarial actors, or notice them if they slip through the cracks.

I do think there are real skills you are pointing to, but to some extent I prefer the world where I don't have those skills and in place of that my allies and I coordinate to identify and exclude people who are using the dark arts.

I like your preference. I'll say some things, but I want to start by emphasizing that I don't think you're making a wrong or bad choice.

I want to talk about what I think the Art could be, kind of for aesthetic reasons. This isn't to assert anything about what you or any given individual should or shouldn't be doing in any kind of moral sense.

So with that said, here are three points:

(1) I think there's a strong analogy here to studying combat and war. Yes, if you can be in a pacifist cluster and just exclude folk who are really into applied competitive strategy, then you have something kind of like a cooperate/cooperate equilibrium. But if that's the whole basis of your culture, it's extremely vulnerable, the way cooperate-bot is vulnerable in prisoners' dilemmas. You need military strength, the way a walled garden needs walls. Otherwise folk who have military strength can just come take your resources, even if you try to exclude them at first.

At the risk of using maybe an unfair example, I think what happened with FTX last year maybe illustrates the point.

Clearer examples in my mind are Ziz and Brent. The point not being "These people are bad!" But rather, these people were psychologically extremely potent and lots of folk in the community could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they'd already had their impact.

Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier. But if you don't do that by studying the Dark Arts, and becoming intimately familiar with them, then what you get is a kind of naïve allergic response that Dark Artists can weaponize.

Again, I don't mean that you in particular or even rationalists in general need to address this. There's nothing wrong with a hobby. I'm saying that as an Art, it seems like rationality is seriously vulnerable if it doesn't include masterful familiarity with the Dark Arts. Kind of like, there's nothing wrong with practicing aikido as a sport, but you're not gonna get the results you hope for if you train in aikido for self-defense. That art is inadequate for that purpose and needs exposure to realistic combat to matter that way.

(2) …and I think that if the Art of Rationality were to include intimate familiarity with the Dark Arts, it would work way way better.

Things like the planning fallacy or confirmation bias are valuable to track. I could stand to improve my repertoire here for sure.

But the most potent forms of distorted thinking aren't about sorting out the logic. I think they look more like reaching deep down and finding ways to become immune to things like frame control.

Frame control is an amazing example in my mind precisely because of the hydra-like nature of the beast. How do you defend against frame control without breaking basic things about culture and communication and trust? How do you make it so your cultural and individual defenses don't themselves become the manual that frame controllers use to get their desired effects?

And this barely begins to touch on the kind of impact that I'd want to call "spiritual". By which I don't mean anything supernatural; I'm talking about the deep psychological stuff that (say) conversing with someone deep in a psilocybin trip can do to the tripper. That's not just frame control. That's something way deeper, like editing someone's basic personality operating system code. And sometimes it reaches deeper even than that. And it turns out, you don't need psychedelics to reach that deep; those chemical tools just open a door that you can open other ways, voluntarily or otherwise, sometimes just by having a conversation.

The standard rationalist defense I've noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/"shame it into oblivion" kind of response. The stuff that disables this immune response is really epistemically strange — things like prefacing with "Here's a fake framework, it's all baloney, don't believe anything I'm saying." Or doing a bunch of embodied stuff to act low-status and unsure. A Dark Artist who wanted to deeply mess with this community wouldn't have to work very hard to do some serious damage before getting detected, best as I can tell (and as community history maybe illustrates).

If this community wanted to develop the Art to actually be skillful in these areas… well, it's hard to predict exactly what that'd create, but I'm pretty sure it'd be glorious. If I think of the Sequences as retooling skeptical materialism, I think we'd maybe see something like a retooling of the best of Buddhist psychotechnology. I think folk here might tend to underestimate how potent that could really be.

(…and I also think that it's maybe utterly critical for sorting out AI alignment. But while I think that's a very important point, it's not needed for my main message for this exchange.)

(3) It also seems relevant to me that "Dark Arts" is maybe something of a fake category. I'm not sure it even forms a coherent cluster.

Like, is being charismatic a Dark Art? It certainly can be! It can act as a temptation. It seems to be possible to cultivate charisma. But the issue isn't that charisma is a Dark Art. It's that charisma is mostly symmetric. So if someone has a few slightly anti-epistemic social strategies in them, and they're charismatic, this can have a net Dark effect that's even strategic. But this is a totally normal level of epistemic noise!

Or how about something simpler, like someone using confirmation bias in a way that benefits their beliefs? Astrology is mostly this. Is astrology a Dark Art? Is talking about astrology a Dark Art? It seems mostly just epistemically hazardous… but where's the line between that and Dark Arts?

How about more innocent things, like when someone is trying to understand systemic racism? Is confirmation bias a helpful pattern-recognizer, or a Dark Art? Maybe it's potentially in service to Dark Arts, but is a necessary risk to learn the patterns?

I think Vervaeke makes this point really well. The very things that allow us to notice relevance are precisely the things that allow us to be fooled. Rationality (and he explicitly cites this — even the Keith Stanovich stuff) is a literally incomputable practice of navigating both Type I and Type II errors in this balancing act between relevance realization and being fooled.

When I think of central examples of Dark Arts, I think mostly of agents who exploit this ambiguity in order to extract value from others.

…which brings me back to point (1), about this being more a matter of skill in war. The relevant issue isn't that there are "Dark Arts". It's that there are unaligned agents who are trying to strategically fool you. The skill isn't to detect a Dark toolset; it's to detect intelligent intent to deceive and extract value.

All of which is to say:

I think a mature Art of Rationality would most definitely include something like skillful navigation of manipulation.
I don't think every practitioner needs to master every aspect of a mature Art. Much like not all cooks need to know how to make a roux.
But an Art that has detection, exclusion, & avoidance as its only defense against Dark Artists is a much poorer & more vulnerable Art. IMO.

See in context

104 Here's the exit.

by Valentine

21st Nov 2022

12 min read

180

104

There's a kind of game here on Less Wrong.

It's the kind of game that's a little rude to point out. Part of how it works is by not being named.

Or rather, attempts to name it get dissected so everyone can agree to continue ignoring the fact that it's a game.

So I'm going to do the rude thing. But I mean to do so gently. It's not my intention to end the game. I really do respect the right for folk to keep playing it if they want.

Instead I want to offer an exit to those who would really, really like one.

I know I really super would have liked that back in 2015 & 2016. That was the peak of my hell in rationalist circles.

I'm watching the game intensify this year. Folk have been talking about this a lot. How there's a ton more talk of AI here, and a stronger tone of doom.

I bet this is just too intense for some folk. It was for me when I was playing. I just didn't know how to stop. I kind of had to break down in order to stop. All the way to a brush with severe depression and suicide.

And it also ate parts of my life I dearly, dearly wish I could get back.

So, in case this is audible and precious to some of you, I'd like to point a way to ease.

The Apocalypse Game

The upshot is this:

You have to live in a kind of mental illusion to be in terror of the end of the world.

Illusions don't look on the inside like illusions. They look like how things really are.

Part of how this one does the "daughter's arm" thing is by redirecting attention to facts and arguments.

"Here's why the argument about AI makes sense."
"Do you have some alternative view of what will happen? How do you address XYZ?"
"What makes it an 'illusion'? I challenge that framing because it dismisses our ability to analyze and understand yada yada."

None of this is relevant.

I'm pointing at something that comes before these thoughts. The thing that fuels the fixation on the worldview.

I also bet this is the thing that occasionally drives some people in this space psychotic, depressed, or into burnout.

The basic engine is:

There's a kind of underlying body-level pain. I would tag this as "emotional pain" but it's important to understand that I really am pointing at physical sensations.
The pain is kind of stored and ignored. Often it arose from a very young age but was too overwhelming, so child-you found methods of distraction.
This is the basic core of addiction. Addictions are when there's an intolerable sensation but you find a way to bear its presence without addressing its cause. The more that distraction becomes a habit, the more that's the thing you automatically turn to when the sensation arises. This dynamic becomes desperate and life-destroying to the extent that it triggers a red queen race.

A major unifying flavor of the LW attractor is intense thought as an addictive distraction. And the underlying flavor of pain that fuels this addiction is usually some variation of fear.
In not-so-coincidental analogy to uFAI, these distracting thoughts can come to form autonomous programs that memetically evolve to have something like survival and reproductive instincts — especially in the space between people as they share and discuss these thoughts with each other.
The rationalist memeplex focuses on AI Ragnarok in part because it's a way for the intense thought to pull fuel from the underlying fear.

In this case, the search for truth isn't in service to seeing reality clearly. The logic of economic races to the bottom, orthogonality, etc. might very well be perfectly correct.

But these thoughts are also (and in some cases, mostly) in service to the doomsday meme's survival.

But I know that thinking of memes as living beings is something of an ontological leap in these parts. It's totally compatible with the LW memeplex, but it seems to be too woo-adjacent and triggers an unhelpful allergic response.

So I suggested a reframe at the beginning, which I'll reiterate here:

Your body's fight-or-flight system is being used as a power source to run a game, called "OMG AI risk is real!!!"

And part of how that game works is by shoving you into a frame where it seems absolutely fucking real. That this is the truth. This is how reality just is.

And this can be fun!

And who knows, maybe you can play this game and "win". Maybe you'll have some kind of real positive impact that matters outside of the game.

But… well, for what it's worth, as someone who turned off the game and has reworked his body's use of power quite a lot, it's pretty obvious to me that this isn't how it works. If playing this game has any real effect on the true world situation, it's to make the thing you're fearing worse.

(…which is exactly what's incentivized by the game's design, if you'll notice.)

I want to emphasize — again — that I am not saying that AI risk isn't real.

I'm saying that really, truly orienting to that issue isn't what LW is actually about.

That's not the game being played here. Not collectively.

But the game that is being played here absolutely must seem on the inside like that is what you're doing.

Ramping Up Intensity

When Eliezer rang the doom bell, my immediate thought was:

"Ah, look! The gamesmaster has upped the intensity. Like preparing for a climax!"

I mean this with respect and admiration. It's very skillful. Eliezer has incredible mastery in how he weaves terror and insight together.

And I don't mean this at all to dismiss what he's saying. Though I do disagree with him about overall strategy. But it's a sincere disagreement, not a "Oh look, what a fool" kind of thing.

What I mean is, it's a masterful move of making the game even more awesome.

(…although I doubt he consciously intended it that way!)

I remember when I was in the thick of this AI apocalypse story, everything felt so… epic. Even questions of how CFAR dealt with garbage at its workshops seemed directly related to whether humanity would survive the coming decades. The whole experience was often thrilling.

And on the flipside, sometimes I'd collapse. Despair. "It's too much" or "Am I even relevant?" or "I think maybe we're just doomed."

These are the two sort of built-in physiological responses to fight-or-flight energy: activation, or collapse.

(There's a third, which is a kind of self-holding. But it has to be built. Infants aren't born with it. I'll point in that direction a bit later.)

In the spirit of feeling rationally, I'd like to point out something about this use of fight-or-flight energy:

If your body's emergency mobilization systems are running in response to an issue, but your survival doesn't actually depend on actions on a timescale of minutes, then you are not perceiving reality accurately.

Which is to say: If you're freaked out but rushing around won't solve the problem, then you're living in a mental hallucination. And it's that hallucination that's scaring your body.

Again, this isn't to say that your thoughts are incorrectly perceiving a future problem.

But if it raises your blood pressure or quickens your breath, then you haven't integrated what you're seeing with the reality of your physical environment. Where you physically are now. Sitting here (or whatever) reading this text.

So… folk who are wringing their hands and feeling stressed about the looming end of the world via AI?

Y'all are hallucinating.

If you don't know what to do, and you're using anxiety to power your minds to figure out what to do…

…well, that's the game.

The real thing doesn't work that way.

But hey, this sure is thrilling, isn't it?

As long as you don't get stuck in that awful collapse space, or go psychotic, and join the fallen.

But the risk of that is part of the fun, isn't it?

(Interlude)

A brief interlude before I name the exit.

I want to emphasize again that I'm not trying to argue anyone out of doing this intense thing.

The issue is that this game is way, way out of range for lots of people. But some of those people keep playing it because they don't know how to stop.

And they often don't even know that there's something on this level to stop.

You're welcome to object to my framing, insist I'm missing some key point, etc.

Frankly I don't care.

I'm not writing this to engage with the whole space in some kind of debate about AI strategy or landscape or whatever.

I'm trying to offer a path to relief to those who need it.

That no, this doesn't have to be the end of the world.

And no, you don't have to grapple with AI to sort out this awful dread.

That's not where the problem really is.

I'm not interested in debating that. Not here right now.

I'm just pointing out something for those who can, and want to, hear it.

Land on Earth and Get Sober

So, if you're done cooking your nervous system and want out…

…but this AI thing gosh darn sure does look too real to ignore…

…what do you do?

My basic advice here is to land on Earth and get sober.

The thing driving this is a pain. You feel that pain when you look out at the threat and doom of AI, but you cover it up with thoughts. You pretend it's about this external thing.

I promise, it isn't.

I know. I really do understand. It really truly looks like it's about the external thing.

But… well, you know how when something awful happens and gets broadcast (like the recent shooting), some people look at it with a sense of "Oh, that's really sad" and are clearly impacted, while others utterly flip their shit?

Obviously the difference there isn't in the event, or in how they heard about it. Maybe sometimes, but not mostly.

The difference is in how the event lands for the listener. What they make it mean. What bits of hidden pain are ready to be activated.

You cannot orient in a reasonable way to something that activates and overwhelms you this way. Not without tremendous grounding work.

So rather than believing the distracting thoughts that you can somehow alleviate your terror and dread with external action…

…you've got to stop avoiding the internal sensation.

When I talked earlier about addiction, I didn't mean that just as an analogy. There's a serious withdrawal experience that happens here. Withdrawal from an addiction is basically a heightening of the intolerable sensation (along with having to fight mechanical habits of seeking relief via the addictive "substance").

So in this case, I'm talking about all this strategizing, and mental fixation, and trying to model the AI situation.

I'm not saying it's bad to do these things.

I'm saying that if you're doing them as a distraction from inner pain, you're basically drunk.

You have to be willing to face the awful experience of feeling, in your body, in an inescapable way, that you are terrified.

I sort of want to underline that "in your body" part a bazillion times. This is a spot I keep seeing rationalists miss — because the preferred recreational drug here is disembodiment via intense thinking. You've got to be willing to come back, again and again, to just feeling your body without story. Notice how you're looking at a screen, and can feel your feet if you try, and are breathing. Again and again.

It's also really, really important that you do this kindly. It's not a matter of forcing yourself to feel what's present all at once. You might not even be able to find the true underlying fear! Part of the effect of this particular "drug" is letting the mind lead. Making decisions based on mental computations. And kind of like minds can get entrained to porn, minds entrained to distraction via apocalypse fixation will often hide their power source from their host.

(In case that was too opaque for you just yet, I basically just said "Your thoughts will do what they can to distract you from your true underlying fear." People often suddenly go blank inside when they look inward this way.)

So instead of trying to force it all at once, it's a matter of titrating your exposure. Noticing that AI thoughts are coming up again, and pausing, and feeling what's going on in your body. Taking a breath for a few seconds. And then carrying on with whatever.

This is slow work. Unfortunately your "drug" supply is internal, so getting sober is quite a trick.

But this really is the exit. As your mind clears up… well, it's very much like coming out of the fog of a bender and realizing that no, really, those "great ideas" you had just… weren't great. And now you're paying the price on your body (and maybe your credit card too!).

There are tons of resources for this kind of direction. It gets semi-independently reinvented a lot, so there are lots of different names and frameworks for this. One example that I expect to be helpful for at least some LWers who want to land on Earth & get sober is Irene Lyon, who approaches this through a "trauma processing" framework. She offers plenty of free material on YouTube. Her angle is in the same vein as Gabor Maté and Peter Levine.

But hey, if you can feel the thread of truth in what I'm saying and want to pursue this direction, but you find you can't engage with Irene Lyon's approach, feel free to reach out to me. I might be able to find a different angle for you. I want anyone who wants freedom to find it.

But… but Val… what about the real AI problem?!

Okay, sure. I'll say a few words here.

…although I want to point out something: The need to have this answered is coming from the addiction to the game. It's not coming from the sobriety of your deepest clarity.

That's actually a complete answer, but I know it doesn't sound like one, so I'll say a little more.

Yes, there's a real thing.

And yes, there's something to do about it.

But you're almost certainly not in a position to see the real thing clearly or to know what to do about it.

And in fact, attempts to figure the real thing out and take action from this drunk gamer position will make things worse.

(I hesitate to use the word "worse" here. That's not how I see it. But I think that's how it translates to the in-game frame.)

This is what Buddhists should have meant (and maybe did/do?) when they talk about "karma". How deeply entangled in this game is your nervous system? Well, when you let that drive how you interact with others, their bodies get alarmed in similar ways, and they get more entangled too.

Memetic evolution drives how that entangling process happens on large scales. When that becomes a defining force, you end up with self-generating pockets of Hell on Earth.

This recent thing with FTX is totally an example. Totally. Threads of karma/trauma/whatever getting deeply entangled and knotted up and tight enough that large-scale flows of collective behavior create an intensely awful situation.

You do not solve this by trying harder. Tugging the threads harder.

In fact, that's how you make it worse.

This is what I meant when I said that actually dealing with AI isn't the true game in LW-type spaces, even though it sure seems like it on the inside.

It's actually helpful to the game for the situation to constantly seem barely maybe solvable but to have major setbacks.

And this really can arise from having a sincere desire to deal with the real problem!

But that sincere desire, when channeled into the Matrix of the game, doesn't have any power to do the real thing. There's no leverage.

The real thing isn't thrilling this way. It's not epic.

At least, not any more epic than holding someone you love, or taking a stroll through a park.

To oversimplify a bit: You cannot meaningfully help with the real thing until you're sober.

Now, if you want to get sober and then you roll up your sleeves and help…

…well, fuck yeah! Please. Your service would be a blessing to all of us. Truly. We need you.

But it's gotta come from a different place. Tortured mortals need not apply.

And frankly, the reason AI in particular looks like such a threat is because you're fucking smart. You're projecting your inner hell onto the external world. Your brilliant mind can create internal structures that might damn well take over and literally kill you if you don't take responsibility for this process. You're looking at your own internal AI risk.

I hesitate to point that out because I imagine it creating even more body alarm.

But it's the truth. Most people wringing their hands about AI seem to let their minds possess them more and more, and pour more & more energy into their minds, in a kind of runaway process that's stunningly analogous to uFAI.

The difference is, you don't have to make the entire world change in order to address this one.

You can take coherent internal action.

You can land on Earth and get sober.

That's the internal antidote.

It's what offers relief — eventually.

And from my vantage point, it's what leads to real hope for the world.

Social & Cultural DynamicsSocial RealityRationalityAICommunity

Frontpage

104

Mentioned in

255Mental Health and the Alignment Problem: A Compilation of Resources (updated April 2023)

180Talking publicly about AI risk

163Slack matters more than any outcome

57Voting Results for the 2022 Review

4Who should write the definitive post on Ziz?

2the gears to ascension

132Duncan Sabien (Deactivated)

2Nicholas / Heather Kross

0Valentine

2Nicholas / Heather Kross

12Nicholas / Heather Kross

3Valentine

1Nicholas / Heather Kross

8ShowMeTheProbability

6ShowMeTheProbability

3Nicholas / Heather Kross

New Comment

181 comments, sorted by

top scoring

Click to highlight new comments since: Today at 4:03 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Richard_Ngo2y*174146

I think there's a bunch of useful stuff here. In particular, I think that decisions driven by deep-rooted fear are often very counterproductive, and that many rationalists often have "emergency mobilization systems" running in ways which aren't conducive to good long-term decision-making. I also think that paying attention to bodily responses is a great tool for helping fix this (and in fact was helpful for me in defusing annoyance when reading this post). But I want to push back on the way in which it's framed in various places as all-or-nothing: exit the game, or keep playing. Get sober, or stay drunk. Hallucination, not real fear.

In fact, you can do good and important work while also gradually coming to terms with your emotions, trying to get more grounded, and noticing when you're making decisions driven by visceral fear and taking steps to fix that. Indeed, I expect that almost all good and important work throughout history has been done by people who are at various stages throughout that process, rather than people who first dealt with their traumas and only then turned to the work. (EDIT: in a later comment, Valentine says he doesn't endorse the claim that people should deal... (read more)

[-]Elizabeth2y6459

it seems that "you need to do the trauma processing first and only then do useful work" is a harmful self-propagating meme in a very similar way as "you need to track and control every variable in order for AI to go well"

This. Trauma processing is just as prone to ouroboros-ing as x-risk work, if not more so.

-2Valentine2y

Agreed. And it's also not actually relevant to my point. (Though I understand why it looks relevant.)

[-]Aiyen2y122

Wouldn't it be relevant in that someone could recognize unproductive, toxic dynamics in their concerns about AI risk as per your point (if I understand you correctly), decide to process trauma first and then get stuck in the same sorts of traps? While "I'm traumatized and need to fix it before I can do anything" may not sound as flashy as "My light cone is in danger from unaligned, high-powered AI and I need to fix that before I can do anything", it's just as capable of paralyzing a person, and I speak both from my own past mistakes and from those of multiple friends.

[-]Valentine2y184

Of course that's possible. I didn't mean to dismiss that part.

But… well, as I just wrote to Richard_Ngo:

If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it's running in you.
Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There's some reason for hope, but it's possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem.
But if it's targeted, the addiction basically doesn't stand a chance.
I'm not trying to say "Just work on traumas and be Fully Healed™ before working on AI risk."
I'm saying something much, much more precise.

I do in fact think there's basically no point in someone working on AI risk if they don't dissolve this specific trauma structure.

Well, or at least make it fully conscious and build their nervous system holding capacity enough that it (a) they can watch it trying to run in real time and (b) they can also reliably stop it from grabbing their inner steering wheel so to speak.

But frankly, for most people it'd be easier just to fully integrate the pain than it would be to develop that level of general nervous system capacity without integrating said pain.

7Valentine2y

I agree. I think we're focusing on different spots. I'm not sure if we actually disagree. The all-or-nothing is with respect to recognizing the illusion. If someone can't even get far enough to notice that their disregulated nervous system is driving an illusion, then what they do is much more likely to create harm than good. That part I totally stand by. There's something of a strawman here in framing what I'm saying as "you need to do the trauma processing first and only then do useful work". I don't think you intended it. Just letting you know, it totally lands for me as a strawman. I am saying that there is some trauma processing (for a person with a system like I'm describing) that absolutely is essential first. But not all of it. I don't know if that's possible, or even a coherent idea.

[-]Richard_Ngo2y*101

I don't understand how specifically you think the process of recognizing the illusion is related to the process of healing traumas. But I also object to ideas like "you need to orient towards your fear as an illusion first and only then do useful work", for roughly the same reasons (in particular, the way it's all-or-nothing). So I'll edit my original comment to clarify that this is a more central/less strawmanny objection.

-3Valentine2y

Okay. I'm not sure what to tell you. This lands for me like "I don't understand how you think turning on the burner is related to the process of cooking the soup." Um… it just is? I already described the mechanisms, so I think the communication gap is somewhere I don't see. I never meant to say or even imply that the fear is an illusion. I was saying that the fear fuels an illusion. And anyone living in such an illusion needs to see through it before they can participate in non-illusion. You can view that as all-or-nothing and therefore objectionable if you like. That's note quite what I mean, but it's not totally wrong. And in this spot I do think there's an "all-or-nothing" truth: If you don't see through an illusion you're in, you can't consciously participate in reality. That lands as almost tautological to me. I didn't need you to do that. But thanks.

2Richard_Ngo2y

I think you interpreted this as incredulity, whereas I meant it as "I don't understand the specific links" (e.g. is recognizing the illusion most of the work, or only a small part? What stops you from healing traumas without recognizing the illusion? etc). I've edited to clarify.

2Valentine2y

Oh, no, I didn't take it as incredulity at all. I'm just honestly not sure why what I'd already said didn't already explain the relationship between trauma healing and seeing through the illusion. I guess I can just say it again in shortened form? For the person design I'm talking about… * There's a pain inside. * There's also a kind of mental/emotional program built around the instruction "Distract from the pain." * Because they can't actually escape the pain, they project it outward through the mind. Which is to say, they create an illusion powered by the pain. * This causes them to think every glimmer of the pain they do notice is about the external thing. The antidote is to look directly at the inner pain & dismantle the "Distract from the pain" program. In practice this requires integrating the pain into consciousness. This is one way of talking about "healing trauma". Once that happens, the program doesn't have a power source anymore. If that doesn't happen and the person insists on focusing on doing things in the world, everything they do will be at least partly in service to distraction rather than solving any real problem. And on the inside they cannot tell the difference between those two without facing the inner pain. So seeing through the illusion isn't cognitive basically at all. To me it's the same thing as trauma processing, for all practical purposes. Does that clarify anything for you? Oh, nothing. If you just go around healing traumas willy-nilly, then you might not ever see through any particular illusion like this one if it's running in you. Kind of like, generically working on trauma processing in general might or might not help an alcoholic quit drinking. There's some reason for hope, but it's possible to get lost in loops of navel-gazing, especially if they never ever even admit to themselves that they have a problem. But if it's targeted, the addiction basically doesn't stand a chance. I'm not trying to say "Just work on

2the gears to ascension2y

I think the word game is in an odd situation here. In game theory parlance, physics is just a game. it's not a question of whether you're in a game; it's what game you interpret yourself to be playing. there are positive-sum games you can interpret yourself to have been playing.

5Valentine2y

Well, I meant to point at something intuitive, and kind of as a nod to Existential Kink. I honestly forgot when writing this piece that "game" has special meaning here. Like with game theory. I just meant to hint at kind of a VR game, or like if you can imagine that the Matrix started out as a game people plugged into but part of the game involves forgetting that you plugged in.

[-]gjm2y144141

If this post had just said "I think some people may feel strongly about AI x-risk for reasons that ultimately come down to some sort of emotional/physical pain whose origins have nothing to do with AI; here is why I think this, and here are some things you can do that might help find out whether you're one of them and to address the underlying problem if so", then I would consider it very valuable and deserving of attention and upvotes and whatnot. I think it's very plausible that this sort of thing is driving at least some AI-terror. I think it's very plausible that a lot of people on LW (and elsewhere) would benefit from paying more attention to their bodies.

... But that's not what this post does. It says you have to be "living in a[...] illusion" to be terrified by apocalyptic prospects. It says that if you are "feeling stressed" about AI risks then you are "hallucinating". It says that "what LW is actually about" is not actual AI risk and what to do about it (but, by implication, this alleged "game" of which Eliezer Yudkowsky is the "gamesmaster" that works by engaging everyone's fight-or-flight reactions to induce terror). It says that, for reasons beyond my understanding, it ... (read more)

-4Valentine2y

Whew! That's a lot. I'm not going to try to answer all of that. In short: I think you're correctly following LW norms. You're right that I wasn't careful about tone, and by the norms here it's good to note that. And also, that wasn't what this piece was about. I intended it as an invitation. Not as a set of claims to evaluate. If you look where I'm pointing, and you recognize some of yourself in it (which it sounds like you don't!), then the suggestions I gesture toward (like Irene Lyon, and maybe loosening the mental grip on the doomy thoughts) might seem worth exploring. I have no intention of putting an argument together, with evidence and statistics and the like, validating the mechanisms I'm talking about. That would actually go in the opposite direction of making an audible invitation. But! I think your contribution is good. It's maybe a little more indignant than necessary. But it's… mmm… fitting, I'll say. I'll leave it at that.

[-]PhilGoetz2y2838

I think it would be more-graceful of you to just admit that it is possible that there may be more than one reason for people to be in terror of the end of the world, and likewise qualify your other claims to certainty and universality.

That's the main point of what gjm wrote. I'm sympathetic to the view you're trying to communicate, Valentine; but you used words that claim that what you say is absolute, immutable truth, and that's the worst mind-killer of all. Everything you wrote just above seems to me to be just equivocation trying to deny that technical yet critical point.

I understand that you think that's just a quibble, but it really, really isn't. Claiming privileged access to absolute truth on LessWrong is like using the N-word in a speech to the NAACP. It would do no harm to what you wanted to say to use phrases like "many people" or even "most people" instead of the implicit "all people", and it would eliminate a lot of pushback.

6gjm2y

(I see that this comment has received a lot of downvotes None of them is from me.)

[-]Duncan Sabien (Deactivated)1y*13297Review for 2022 Review

I think this post is emblematic of the problem I have with most of Val's writing: there are useful nuggets of insight here and there, but you're meant to swallow them along with a metric ton of typical mind fallacy, projection, confirmation bias, and manipulative narrativemancy.

Elsewhere, Val has written words approximated by ~"I tried for years to fit my words into the shape the rationalists wanted me to, and now I've given up and I'm just going to speak my mind."

This is what it sounds like when you are blind to an important distinction. Trying to hedge magic things that you do not grok, engaging in cargo culting. If it feels like tediously shuffling around words and phrases that all mean exactly the same thing, you're missing the vast distances on the axis that you aren't perceiving.

The core message of "hey, you might well be caught up in a false narrative that is doing emotional work for you via providing some sense of meaning or purpose and yanking you around by your panic systems, and recognizing that fact can allow you to do anything else" is a good one, and indeed it's one that many LessWrongers need. It's even the sort of message that needs some kind of shock along wi... (read more)

9Valentine2mo

I've been reflecting on this since it was posted. Coming back to it from time to time. I just wanted to make a note saying: received. I believe I see your point, and I've been taking it in. (I also disagree with some of the narrativemancy you employ here about me. It's difficult for me to publicly agree with anything in your comment here because of a similar mechanism that you're objecting to in the OP. I wish I didn't need to add this note. I'd rather just say "I'm hearing something meaningful in what you're saying and have been taking it in." That's the part that matters. But I can do so only if I also register that I very much disagree with your model of what I was trying to do, and in some cases I strongly disagree with your analysis of what I was in fact doing. Thankfully I can still learn from your message anyway. I just wanted to say — more to LW than to you, really — that I've heard your objection and have been taking the truth I can find in it seriously.)

6Bohaska1y

Are there any similar versions of this post on LW which express the same message, but without the patronising tone of Valentine? Would that be valuable?

1Espedair Street2mo

ICYMI - Kaj mentioned https://www.lesswrong.com/posts/byewoxJiAfwE6zpep/reality-revealing-and-reality-masking-puzzles in a comment below - I think Anna does a pretty good job of that in that post

[-]jimrandomh2y115108

I'm sure there are many people whose inner experience is like this. But, negative data point: Mine isn't. Not even a little. And yet, I still believe AGI is likely to wipe out humanity.

[-]leogao2y279

Seconded: mine also isn't.

Also, for what it's worth, I also don't think of myself as the kind of person to naturally gravitate towards the apocalypse/"saving the world" trope. From a purely narrative-aesthetic perspective, I much prefer the idea of building novel things, pioneering new frontiers, realizing the potential of humanity, etc, as opposed to trying to prevent disaster, reduce risk, etc. I am quite disappointed at reality for not conforming to my literary preferences.

[-]Adam Zerner2y121

It's interesting how people's responses can be so different here. I'm someone who gets pretty extreme anxiety from the x-risk stuff, at least when I'm not repressing those feelings.

[-]Valentine2y115

Yep. That just means this wasn't written for you! I expect this wasn't written for a lot of (most?) people here.

[-]mako yass2y4026

I really wish that the post has been written in a way that let me figure out it wasn't for me sooner...

I think it would have saved a lot of time if the paragraph in bold had been at the top.

[-]Vaniver2y156

I came here to say something roughly like Jim's comment, but... I think what I actually want is grounding? Like, sure, you were playing the addictive fear game and now think you're out of it. But do you think I was? If you think there's something that differentiates people who are and aren't, what is it?

[Like, "your heart rate increases when you think about AI" isn't a definitive factor one way or another, but probably you could come up with a list of a dozen such indicators, and people could see which are true for them, and we could end up with population statistics.]

[-]Kaj_Sotala2y4314

I think that at least the kinds of "Singularity-disrupted" people that Anna describes in "Reality-Revealing and Reality-Masking Puzzles" are in the fear game.

Over the last 12 years, I’ve chatted with small hundreds of people who were somewhere “in process” along the path toward “okay I guess I should take Singularity scenarios seriously.” From watching them, my guess is that the process of coming to take Singularity scenarios seriously is often even more disruptive than is losing a childhood religion. Among many other things, I have seen it sometimes disrupt:
People's belief that they should have rest, free time, some money/time/energy to spend on objects of their choosing, abundant sleep, etc.
“It used to be okay to buy myself hot cocoa from time to time, because there used to be nothing important I could do with money. But now—should I never buy hot cocoa? Should I agonize freshly each time? If I do buy a hot cocoa does that mean I don’t care?”
People's in-practice ability to “hang out”—to enjoy their friends, or the beach, in a “just being in the moment” kind of way.
“Here I am at the beach like my to-do list told me to be, since I’m a good EA who is planning not to burn out. I’ve g

... (read more)

5Valentine2y

I honestly don't know. I lean toward no? But don't believe me too much there. The main one I'm interested in is "Do you recognize yourself in the dynamic I spelled out?" I like Kaj bringing in that list. I think that's helpful. A lot of how I pick this stuff out isn't a mental list. There's a certain rushedness. A pressure to their excitement about and fixation on doomy things. Conversation flows in a particularly squeezy and jagged way. Body movements are… um… fitting of the pattern. :-P There was a noticeable surge of this when Scott came out with "Meditations on Moloch". I remember how at the EAG that year a bunch of people went and did a mock magical ceremony against Moloch. (I think Scott published it right as EAG was starting.) That totally had the energy of the thing I'm talking about. Playful, but freaked out. I know this doesn't help with the statistics thing. But I'm way less confident of "These are the five signs" than I am about this feeling tone.

4Shiroe1y

Same. I feel somewhat jealous of people who can have a visceral in-body emotional reaction to X-risks. For most of my life I've been trying to convince my lizard brain to feel emotions that reflect my beliefs about the future, but it's never cooperated with me.

[-]Elizabeth2y7651

AFAICT from skimming, the object level of this post has a lot of overlap with my own algorithm. I limit engagement with x-risk to an amount that's healthy and sustainable for me. I keep non-x-risk clients in part to ground me in the real world. I'm into trauma processing and somatics. I think the fact that the people most scared of AGI risk are also the ones most scared of not developing AGI should raise some eyebrows. I treat "this feels bad" as a reason to stop without waiting for a legible justification.

And right now I'm using that last skill to not read this post. I wouldn't have even skimmed if I didn't think it was important to make this comment and have it not be totally uninformed. When I read this I feel awful, highly activated, and helpless/freeze response. It instills the same "you can't trust yourself, follow this rigidity" that it's trying to argue against.

You can't fight fire with fire, getting out of a tightly wound x-risk trauma spiral involves grounding and building trust in yourself, not being scared into applying the same rigidity in the opposite direction.

9Richard_Ngo2y

Very nice observation.

[-]gjm2y146

I agree, but how sure are we that it's actually a fact?

[EDITED to add:] One not-particularly-sinister-or-embarrassing possible explanation, if it is true, is that both are driven by a single underlying issue: how capable does any given person expect AGI to be? Imagine someone in WW2 thinking about whether to develop nuclear weapons. It seems plausible that { people who think it's super-vital to do it because whoever does it will win the war for sure } and { people who think it's super-dangerous to do it because these weapons could do catastrophic damage } might be roughly the same set of people.

3alkjash2y

The comment is generally illuminating but this particular sentence seems too snappy and fake-wisdomy to be convincing. Would you mind elaborating?

[-]Kaj_Sotala2y5921

There's a class of things that could be described as losing trust in yourself and in your ability to reason.

For a mild example, a friend of mine who tutors people in math recounts that many people have low trust in their ability to mathematical reasoning. He often asks his students to speak out loud while solving a problem, to find out how they are approaching it. And some of them will say something along the lines of, "well, at this point it would make the most sense to me to [apply some simple technique], but I remember that when our teacher was demonstrating how to solve this, he used [some more advanced technique], so maybe I should instead do that".

The student who does that isn't trusting that the algorithm of "do what makes the most sense to me" will eventually lead to the correct outcome. Instead, they're trying to replace it with "do what I recall an authority figure doing, even if I don't understand why".

Now it could be that the simple technique is wrong to apply here, and the more advanced one is needed. But if the student had more self-trust and tried the thing that made the most sense to them, then their attempt to solve the problem using the simple approach might help ... (read more)

5TekhneMakre2y

IDK if helpful, but my comment on this post here is maybe related to fighting fire with fire (though Elizabeth might have been more thinking of strictly internal motions, or something else): https://www.lesswrong.com/posts/kcoqwHscvQTx4xgwa/?commentId=bTe9HbdxNgph7pEL4#comments And gjm's comment on this post points at some of the relevant quotes: https://www.lesswrong.com/posts/kcoqwHscvQTx4xgwa/?commentId=NQdCG27BpLCTuKSZG

5Elizabeth2y

That's a super reasonable request that I wish I was able to fulfill. Engaging with Val on this is extremely costly for me, and it's not reasonable to ask him to step out of a conversation on his own post, so I can't do it here. I thought about doing a short form post but couldn't feature creeped myself to the point it was infeasible.

[-]Valentine2y132

…it's not reasonable to ask [Val] to step out of a conversation on his own post…

If it's understood that I'm not replying because otherwise the contribution won't happen at all rather than because I have nothing to say about it, then I'm fine stepping back and letting you clarify what you mean. If that helps.

4alkjash2y

Sure, no big deal.

-4Valentine2y

I think you're reacting to a tone, not content. I absolutely do not mean anything like "Don't trust yourself, trust this instead." I never mean anything like that. I do mean not to trust the mind though. And for reasons you can come to see for yourself. And people who are highly, highly fused with their minds can't easily tell the difference. That said, I applaud the algorithm. Something not feeling good is great reason not to engage with it. Just remember that that's a fact about your relationship to the thing, not a fact about the thing.

[-]Elizabeth2y2211

I think you're reacting to a tone, not content.

Yes, this is correct.

[-]TekhneMakre2y4231

Neither up- nor down-voted; seems good for many people to hear, but also is typical mind fallacying / overgeneralizing. There's multiple things happening on LW, some of which involve people actually thinking meaningfully about AI risk without harming anyone. Also, by the law of equal and opposite advice: you don't necessarily have to work out your personal mindset so that you're not stressed out, before contributing to whatever great project you want to contribute to without causing harm.

-19Valentine2y

[-]AnnaSalamon2y341

As a personal datapoint: I think the OPs descriptions have a lot in common with how I used to be operating, and that I think this would have been tremendously good advice for me personally, both in terms of its impact on my personal wellness and in terms of its impact on whether I did good-for-the-world things or harmful things.

(If it matters, I still think AI risk is a decent pointer at a thingy in the world that may kill everyone, and that this matters. The "get sober" thing is a good idea both in relation to that and broadly AFAICT.)

4Valentine2y

Thank you for adding your personal data point. I think it's helpful in the public space here. But also, personally I liked seeing that this is (part of) your response. I totally agree.

[-]janus2y*3414

This is beautifully written and points at what I believe to be deep truths. In particular:

Your brilliant mind can create internal structures that might damn well take over and literally kill you if you don't take responsibility for this process. You're looking at your own internal AI risk.
...
Most people wringing their hands about AI seem to let their minds possess them more and more, and pour more & more energy into their minds, in a kind of runaway process that's stunningly analogous to uFAI.

But I won't say more about this right now, mostly because I don't think I can do it justice with the amount of time and effort I'm prepared to invest writing this comment. On that note, I commend your courage in writing and posting this. It's a delicate needle to thread between many possible expressions that could rub people the wrong way or be majorly misinterpreted.

Instead I'll say something critical and/or address a potential misinterpretation of your point:

What is this sobriety you advocate for?

I'm concerned that sobriety might be equivocated with giving in to the cognitive bias toward naive/consensus reality. In a sense of the word, that is what "sobriety" is: a balance of cognitive h... (read more)

[-]Valentine2y256

What is this sobriety you advocate for?

Ah, I'm really glad you asked. I tried to define it implicitly in the post but I was maybe too subtle.

There's this specific engine of addiction. It's the thing that distracts without addressing the cause, and becomes your habitual go-to for dealing with the Bad Thing. That creates a feedback loop.

Sobriety is with respect to an addiction. It means dropping the distraction and facing & addressing the thing you'd been previously distracting yourself from, until the temptation to distract yourself extinguishes.

Alcohol being a seed example (hence "sobriety"). The engine of alcoholism is complex, but ultimately there's an underlying thing (sometimes biochemical, but very often emotional) that's a sensation the alcoholic's mind/body system has identified as "intolerable — to be avoided". Alcohol is a great numbing agent and can create a lot of unrelated sensations (like dizziness), but it doesn't address (say) feelings of inadequacy.

So getting sober isn't just a matter of "don't drink alcohol", but of facing the things that drive the impulse to reach for the bottle. When you extinguish the cause, the effect evaporates on its own — modulo habits.

I... (read more)

4janus2y

Now that you've explained this seems obviously the right sense of sobriety given the addiction analogy. Thank you!

3Valentine2y

Quite welcome.

[-]Ben2y2614

Interesting take. I haven't seen this happen on AI, but I do know two people who have an environmentalism fear spiral thing. My diagnosis was very different: I think the people I know actually have anxiety, or panic attacks or similar for mental health reasons. The environmentalism serves as camouflage. Thought 1: "Why am I depressed/anxious/whatever when things in my life are pretty good?" Then, instead of Thought 2 being "Maybe I should talk to a friend/do something that might cheer me up/see a doctor" they instead get thought 2 "Oh, its because humanity is going to destroy the world and everything will be awful. Man, its great that I am such a well-adjusted, big-picture, caring person that giant planet-scale forces that barely effect me personally have more impact on my emotional state than the actual day to day of my own life." Not only does the camo prevent them addressing the real problem (Ok, the environment is a real problem, but its not the only problem, and its not the problem they are suffering from at the moment), but it also weaponizes all kinds of media against themselves.

[-]Dagon2y2423

Upvoted because it's pointing at a real source of pain, and it's very good to talk about. But I suspect there's a lot of typical mind fallacy in the parts that sound more universal and less "here's what happened to and worked for me".

For me, I went through my doomsday worries in my teens and twenties, long before AI was anything to take seriously. Nuclear war or environmental collapse (or one causing the other) were assumed to be the forms of destruction to expect. Over the course of a decade or two, I was able to accept that, for me, "memento mori" was the root of the anxiety. I don't want to die, and I probably will anyway. There may be no actual outside meaning to my life, or by extension to anyone else's. And that doesn't prevent me from caring about other people (both individuals and groups, though not by any means equally), nor about my own experiences. These are important, even if they're only important to me (and, I hope, to some other humans).

5Valentine2y

In parts of this I'm talking to the kind of person who could benefit from being spoken to about this. My experience is that folk who need support out of tough spots like this have a harder time hearing the deeper message when it's delivered in carefully caveated epistemically rigorous language. It shoves them too hard into thinking, and usually in ways that activate the very machinery they're trying to find a way to escape. I know that's a little outside the discourse norms of LW. Caveating things not as "People experience X" but instead as "I experienced X, and I suspect it's true of some others too". I totally respect that has a place here. Just not so much when trying to point out an exit. I like you sharing your experience overview here. Thank you. I resonate with a fair bit of it, though I came at it from a really different angle. (I grew up believing I'd live forever, then "became mortal" at 32. Spent a few years in nihilistic materialist hell. A lot of what you're saying reminds me of what I was grappling with in that hell. Now that's way, way more integrated — but probably not in a way the LW memeplex would approve of.)

[-]janus2y*112

I lived in "nihilistic materialist hell" from the ages of 5 (when it hit me what death meant) and ~10. It -- belief in the inevitable doom of myself and everyone I cared for and ultimately the entire universe to heat death -- was at times directly apprehended and completely incapacitating, and otherwise a looming unendurable awareness which for years I could only fend off using distraction. There was no gamemaster. I realized it all myself. The few adults I confided in tried to reassure me with religious and non-religious rationalizations of death, and I tried to be convinced but couldn't. It was not fun and did not feel epic in the least, though maybe if I'd discovered transhumanism in this period it would've been a different story.

I ended up getting out of hell mostly just by developing sufficient executive function to choose not to think of these things, and eventually to think of them abstractly without processing them as real on an emotional level.

Years later, I started actually trying to do something about it. (Trying to do something about it was my first instinct as well, but as a 5 yo I couldn't think of anything to do that bought any hope.)

But I think the machinery I... (read more)

9philh2y

I kinda feel like my reaction to this is similar to your reaction to frames: To be more explicit, I feel like... sure, I can believe that sometimes epistemic rigor pushes people into thinky-mode and sometimes that's bad; but epistemic rigor is good anyway. I would much prefer for people to get better at handling things said with epistemic rigor, than for epistemic rigor to get thrown aside. And maybe that's not realistic everywhere, but even then I feel like there should be spaces where we go to be epistemically rigorous even if there are people for whom less rigor would sometimes be better. And I feel like LessWrong should be such a space. I think the thing I'm reacting to here isn't so much the lack of epistemic rigor - there are lots of things on LW that aren't rigorous and I don't think that's automatically bad. Sometimes you don't know how to be rigorous. Sometimes it would take a lot of space and it's not necessary. But strategic lack of epistemic rigor - "I want people to react like _ and they're more likely to do that if I'm not rigorous" - feels bad.

2Valentine2y

That's not what I meant. I mean this much more like switching to Spanish when speaking with a Mexican store clerk. We can talk about the virtues of English all we want to, and maybe even justify that we're helping the clerk deepen their skill with interfacing with the modern world… but really, I just want to communicate. You can frame that as dropping standards in order to have a certain effect on them, but that's a really damn weird frame.

3philh2y

I think this relies on "Val is not successfully communicating with the reader" being for reasons analogous to "Val is speaking English which the store clerk doesn't, or only speaks it poorly". But I suspect that if we unpacked what's going on, I wouldn't think that analogy held, and I would still think that what you're doing seems bad. (Also, I want to flag that "justify that we’re helping the clerk deepen their skill with interfacing with the modern world" doesn't pattern match to anything I said. It hints at pattern matching with me saying something like "part of why we should speak with epistemic rigor is to help people hear things with epistemic rigor", but I didn't say that. You didn't say that I did, and maybe the hint wasn't intentional on your part, but I wanted to flag it anyway.)

[-]drethelin2y2310

I think this essay is blatantly manipulative bullshit written in a deliberately hypnotic style, that could be modified to target any topic anyone cares about.

[-]ztzuliios2y121

It does strike me as a rather fully general counterargument, written in a deliberately obfuscatory/"woo" style. The focus on "listening to your body" seems like an obfuscation, it's an appeal to something deliberately put beyond measurement. This does seem like it could apply to anything anyone cares about (you're a Red Sox fan? You're addicted to the suffering, your body is telling you to stop, land on Earth and get sober!). If you have any reasons to disagree, that's coming from a place of addiction and you need to stop caring and presumably follow a similar life-path to OP because that is the only thing that works, everything else is a death-cult.

I don't buy it, to say the least, and I think it's only the social connections that people have to the OP that make anyone treat it charitably. People have been saying this since the earliest days of the discussion of this topic on the Internet; this fully general counterargument predates Eliezer Yudkowsky being appropriately pessimistic about AI.

I also think that the characterization that all rationalism comes from "disembodiment" is essentially an ableist slur. Using ableist slurs and appealing to t... (read more)

3Slider2y

I think the idea of listening to your body is actually to make visible and thus measurable atleast on the inside the thing. It kinda does require a good faith approach. The hope is that people that are alexithymic might not be (coining words here) asomathymic, that people that do not have verbal access to their emotions (to a sufficient degree) would be able to have bodily access to them (to a sufficient degree). Assumptions that internal emotional access is easy and resonable to expect might be improperly ablist. But it can also be taken in the sense that emotional access is not assumed nor taken to be easy and "try wider spectrum of emotional access" is an action that would not and should not be done unprompted. Giving an advice of "have you tried to switch it off and on again?" does not neccesarily comment on the sophistication of interventions tried.

6ztzuliios2y

The problem isn't that access to emotion is ableist. I think that suggestion is itself ableist, neurodiverse people have complete access to their emotions, their emotional reactions to certain things might simply be different. The problem is that no matter what you do, if you come to a conclusion different from OP, you are simply still "disembodied." You just need to "do more work." This is a way of counting the hits and excusing the misses. "Embodiment" is not "being in touch with your emotions," it is acting in the manner prescribed. What is ableist is saying that there is a single state, "embodiment," which coincidentally overlaps entirely with several other things prescribed, and if you are not in that state, there is a psychological problem with you. This is neurotypical supremacy. As I said in the other post in this thread to which you replied, there are other ways to deal with this. You do not have to do breathwork. You do not have to meditate. You do not have to "listen to your body." These are ideological prescriptions. They poorly emulate cognitive-behavioral therapy, which is a much more effective way to process emotions and resolve maladaptive behavior patterns. This is why the comment parent and myself think that this post is manipulative. It presents a real problem, but frames it in terms such that the only possible solution is the wholesale adoption of the author's ideology. The honest post on this topic would have mentioned other solutions, which maybe the author did not personally experience but understands, through systematizing and integrating their own experiences and the experiences of others, to be also solutions to the same problem.

2Slider2y

I understood it as a method of getting an access to emotions. The problem framing does not really carry an interpretation where you could be 100% aware of everything and still be suffering from the problem, because the antidote offered is to become aware of something (100% awereness might be superhumanly difficult). Claiming that most blind people do not see well 20 meters away is not disparaging in itself. Alexithymia is a catalogued autism trait. It is a spectrum and when you have met one autist you have met one autist. So while assuming all traits upon learning one of them would be erroneuos, the presence of each of the traits become relevant. It is sensible to check whether a particular blind person can see well 1 meter away, is able to turn their eyeballs or knows how to echolocate. Poor understanding of autism can lead to treating disparaging properties to be autism traits. Even misrepresenting frequency can have the same effect. Special interests are a thing but deducing "autistic -> spends daily 3 hours on some specific topic" is ignorantly wrong. Alexithymias basedness as a trait is not very questionable. As a trait alexithymia directly deals with awereness (it is not athymia in the same go). Thus lack of awereness is relevant to alexithymia. So to think without knowing that in the intersection of "awereness" and "autism" alexithymia is worth processing is a leap that can be justified in good faith. Thus I disagree and think that "suggesting that access to emotion is ablist" is not ablist. Being demanding and making a typical mind fallacy is quite bad a combo. Being sure that the antidote has high reliability does commit that kind of bad. I do think that insisting that it doesn't work is ignoring that alexithymic people can respond to stuff like this positively, to project a particular responce profile to be typical to the point of fallacy. Selling a placebo and a dangerously unreliable drug are slightly different things. The post does admit guilt of be

-3Valentine2y

Okay, I'm mostly fine with you two having your exchange and me mostly ignoring it, but I'm gonna speak up against this bit: No. That's not what I said and it's not what I meant. You're making that part up. I'm describing a structure. It doesn't have a damn thing to do with convincing people of something. It's about pointing at reality and inviting people to see what I'm pointing at. If you don't want to look, or you look and you see something else, that's fine by me. Honestly. I doubt my saying this has a damn effect on your sense of what I am or am not saying or intending, honestly. But I'm not going to just let this calibre of bullshit projection slide by without comment.

1ztzuliios2y

How is this falsifiable? Can you point to five people who have done this, but still have a different orientation from you?

[-]Eli Tyre2y*180

Poll: Does your personal experience resonate with what you take Val to be pointing at in this post?

Options are sub-comments of this parent.

Please vote by agreeing, not upvoting, with the answer that feels right to you. Please don't click the disagree button for options you disagree with, so that we can easily tabulate numbers by checking how many people have voted.

(Open to suggestions for better ways to set up polls, for the future.)

2Eli Tyre2y

Maybe something like what Val is pointing at is true of me, but I'm not sure.

2Eli Tyre2y

I don't resonate with what I take Val to be pointing at here.

1Eli Tyre2y

I personally resonate with what I take Val to be pointing at here.

[-]bugsbycarlin2y1810

A helpful tool on the way to landing and getting sober is exercise. Exercise is essentially a displacement, like any of the other addictions, but it has the unique and useful feature that it processes out your chemicals, leaving you with less stress chemicals in circulation, and a refractory period before your body can make more.

Almost no matter your physical capabilities, there is something you can go do that makes you sweat and tires you out... and breaks the stress-focus-stress-focus cycle.

Edit: btw, this is great stuff, very good for this community to name it and offer a path away.

Related, but addressing a very different side of the AI risk mindset: https://idlewords.com/talks/superintelligence.htm

[-]Vanessa Kosoy2y174

Personally, I sometimes have the opposite metacognitive concern: that I'm not freaking out enough about AI risk. The argument goes: if I don't have a strong emotional response, doesn't it mean I'm lying to myself about believing that AI risk is real? I even did a few exercises in which I tried to visualize either the doom or some symbolic representation of the doom in order to see whether it triggers emotion or, conversely, exposes some self-deception, something that rings fake. The mental state that triggered was interesting, more like a feeling of calm meditative sadness than panic. Ultimately, I think you're right when you say, if something doesn't threaten me on the timescale of minutes, it shouldn't send me into fight-or-flight. And, it doesn't.

I also tentatively agree that it feels like there's something unhealthy in the panicky response to Yudkowsky's recent proclamation of doom, and it might lead to muddled thinking. For example, it seems like everyone around here are becoming convinced of shorter and shorter timelines, without sufficient evidence IMO. But, I don't know whether your diagnosis is correct. Most of the discourse about AI risk around here is not producing any real progress on the problem. But, occasionally it does. And I'm not sure whether the root of the problem is psychological/memetic (as you claim) or just that it's a difficult problem that only a few can meaningfully contribute to.

4Valentine2y

Just to be clear, I'm not talking about strong emotional responses per se. I'm talking about the body freaking out — which often produces strong emotions. I'm way less concerned about heart-wrenching grief than I am about nervousness, for instance. That's fair. Though I do think the immense difficulty with coordination around AI risk stuff totally is a memetic thing, and that AI risk is a hard enough problem that a focus on tackling it directly with what amounts to a shrug toward the memetic problem is kind of pushing the door on its hinges.

2Nicholas / Heather Kross2y

There are a few different psychological theories about how emotions get produced, and how much other physical reactions influence and/or are influenced by that. So... this isn't a particularly useful distinction, and I didn't see much of it in-depth in the post proper.

0Valentine2y

If this wasn't a useful distinction for you, then why comment on it? To tell me not to have made it at all?

2Nicholas / Heather Kross2y

Good point, just something I noticed, but now that you mention it it's not very useful. EDIT: wait, no, I was commenting on it to point out that you don't seem to have made the distinction yourself in the post proper.

2Mitchell_Porter2y

It's easier to be more composed about a problem, when you think you have the kernel of a solution. I mean, aren't you the founder of the Infra-Bayesian school of thought?

[-]mkualquiera2y132

I am very conflicted about this post.

On the one hand it deeply resonates with my own observations. Many of my friends from the community seem to be stuck on the addictive loop of proclaiming the end of the world every time a new model comes out. I think it's even more dangerous, as it becomes a social activity: "I am more worried than you about the end of the world, because I am smarter/more agentic than you, and I am better at recognizing the risk that this represents for our tribe." gets implicitly tossed around in a cycle where the members keep trying to one-up each other. This only ends when their claims get so absurd as to say the world will end next month, but even this absurdity seems to keep getting eroded over time.

Like someone else said here in the comments, if was reading about this issue in some unrelated doomsday cult from a book, I would immediately dismiss them as a bunch of lunatics. "How many doomsday cults have existed in history? Even if yours is based on at least some solid theoretical foundations, what happened to the previous thousands of doomsday cults that also were, and were wrong?"

On the other hand I have to admit that the arguments in your pos... (read more)

3Valentine2y

Um. That's a thing I suppose someone could do with some variation of these frames, sure. That's not a move I'm at all interested in though. I really would prefer no one does this. It warps the point into something untrue and unkind. I'm much more interested in something like: * There's this specific internal system design a person can fall into. * It's a pretty loud feature of the general rationalist cluster. * If you (a general reader, not you mkualqulera per se) are subject to this pattern and you want out, here's a way out. * Also, people who are in such a pattern but don't want out (or are too stuck in it to see they're in it) are in fact making the real thing harder to solve. So noticing and getting out of this pattern really is a priority if you care about the real thing. Now, if someone freaks out at me for pointing this out and makes some bizarre assumptions about what I'm saying (like, say, that I'm claiming there's no AI problem or that I'm saying any action to deal with it is delusional), at that point I consider it way more likely that they're "drunk", and I'm much more likely to ignore what they have to say. Their ravings and condemnation land for me like a raging alcoholic who's super pissed I implied they have a problem with an addiction. But none of this is about me winning arguments with people. It's about pointing out a mechanism for those who want to see it. And for those for whom it doesn't apply, or to whom it does but they're determined not to look? Well, cool, good on them! Truly. (Also, I like the kind of conflict you're wrestling with. I don't want to try to argue you out of that. I just wanted to clarify this part a bit.)

-7[anonymous]2y

[-]Slimepriestess2y132

Thank you so much for writing this. I wish I had this in 2018 when I was spiraling really badly. I feel like I only managed to escape from the game by sheer luck and it easily could have killed me, hell it HAS killed people. Not everyone manages to break in a way that breaks them out of game and not just obliterate them.

I wrote a story about my attempts to process through a lot of this earlier this year
https://voidgoddess.org/2022/11/15/halokilled/

[-]Ben Schwyn2y139

While some people might be doing intense thinking / writing, others like myself are distracting themselves via intense listening/perceiving/reading --- covering up their own thoughts and cares by taking in lots of information and sedating/overwhelming their emotions.

[-]Nicholas / Heather Kross2y122

This seems... testable? Like, it's kind of the opposite message of Yudkowsky's "try harder" posts.

Have two groups work on a research problem. One is in doom mode, one is in sober mode. See which group makes more progress.

3Valentine2y

Yep. I don't like your proposed test (what's going to define "progress"?), but yes. My main purpose for this post wasn't to make amazing AI safety researchers though. It was to offer people who want out of the inner doomsday trap a way of exiting. That part is a little more tricky to test. But if someone wants to test it and wants to put in the effort of designing such a test, I think it's probably doable.

1Nicholas / Heather Kross2y

Yeah, the test has to be set up with all the normal caveats in advance (including being specific enough to measure, but broad enough to avoid people having good excuses to ignore whatever its conclusions turn out to be).

[-]MikkW2y*92

Uh, no.

Maybe I just genuinely care about not having terrible things happen to me and everyone else in the world? There's no game there, no broken addiction mechanisms inside.

I strong-downvoted this. [edit: I removed a statement about my feelings in reaction to this, that I feel was a little too much]

I just want to do what I can to keep the people I love from dying.

0Valentine2y

I guess you missed the part that I repeated several times that I wasn't saying there isn't something real to address. And the interlude. And the request for same-sided exploration. Alas.

3MikkW2y

I didn't miss that. That doesn't change what the rest of your post objectively is saying. (Well, I did overlook the bit about same-sided exploration. But idk, the way this post was worded kinda kills my desire to do that) Edit: as far as the interlude, it only makes sense given the flawed thesis that is the precise thing I'm reacting negatively to.

[-]PaulK2y90

I think your diagnosis of the problem is right on the money, and I'm glad you wrote it.

As for your advice on what a person should do about this, it has a strong flavor of: quit doing what you're doing and go in the opposite direction. I think this is going to be good for some people but not others. Sometimes it's best to start where you are. Like, one can keep thinking about AI risk while also trying to become more aware of the distortions that are being introduced by these personal and collective fear patterns.

That's the individual level though, and... (read more)

5PaulK2y

I also think that the fact that AI safety thinking is so much driven by these fear + distraction patterns, is what's behind the general flail-y nature of so much AI safety work. There's a lot of, "I have to do something! This is something! Therefore, I will do this!"

3Valentine2y

I agree… and also, I want to be careful of stereotypes here. Like, I totally saw a lot of flail nature in what folk were doing when I was immersed in this world years ago. But I also saw a lot of faux calmness and reasonableness. That's another face of this engine. And I saw some glimmers of what I consider to be clear lucidity. And I saw a bunch that I wasn't lucid enough at the time to pay proper attention to, and as such I don't have a clear opinion about now. I just lack data because I wasn't paying attention to the people standing in front of me. :-o But with that caveat: yes, I agree.

2Valentine2y

I mostly just agree. I hesitate to ever give a rationalist the advice to keep thinking about something that's causing them to disembody while they work on embodiment. Even if there's a good way for them to do so, my impression is that most who would be inclined to try cannot do that. They'll overthink. It's like suggesting an alcoholic not stop cold-turkey but leaving them to decide how much to wean back. But I do think there's a balance point that if it could be enacted would actually be healthier for quite a few people. I'm just not holding most folks' hands here! So the "cold turkey" thing strikes me as better general advice for those going at it on their own with minimal support.

[-]Elizabeth1y82Review for 2022 Review

I stand by what I said here: this post has a good goal but the implementation embodies exactly the issue it's trying to fight.

[-]Viliam2y80

Valentine wrote an important message in a metaphorical language that will rub some people the wrong way (that includes me), but it seems like the benefit for those who need to hear it may exceed the annoyance of those who don't. Please let's accept it this way, and not nitpick the metaphors.

As a boring person, I would prefer to have a boring summary on the top, or maybe something like this:

If X is freaking you out, it is a fact about you, not about X. Read how this applies to the topic "AI will kill you"...

The longer boring version is the following: Human ... (read more)

2Valentine2y

This is really good. Thank you. I'd add that there's a very specific structure I'm trying to point at. Something I think is right to call an addiction, and a pathway out of said addiction. I'm pretty sure that could be said in detail in a "boring" way too. I just really suck at creating "boring" versions of things. :-D Thank you for this.

2Viliam2y

In Transactional Analysis there is something called "racket" (not mentioned on its Wikipedia page), a concept that people have their habitual emotion... not meaning that they like it or approve of it, just that for many things that happen they will find an excuse to translate them to that emotion. As usual, the psychoanalytical explanation is that your parents paid to you attention in childhood when you exhibited that emotion, and ignored you when you exhibited other emotions. Thus, converting every experience to given emotion is how you unconsciously pay for being paid attention to.

[-]Noosphere892y84

I'm very conflicted about this post. On the one hand, many of it's parts are necessary things for LWers to hear, and I'm getting concerned about the doom loop that seems to form a cult-like mentality on AI.

On the other hand, it also has serious issues in it's framing, and I'm worried that the post is coming out of a mentality that isn't great as well.

[-]Ben Pace2y7-9

Great post, thanks for writing it.

I try to reward posts I like with thoughtful commentary/disagreement, but there's a sense in which this post doesn't want to continue an existing spiraling thought pattern, it wants me to go out and do whatever I want to after putting that down.

[-]Ben Pace2y235

After reading the other comments, I'll at least add in the datapoint that I have experienced a ton of "ruminating-about-AI-risk-strategy-as-escapism" in my life, and being able to not do that has been a pretty key step in actually making progress on the problem.

When I remember back to those times when I was trapped in it (not saying I don't still indulge from time to time), I think I would have found this post quite scary to engage with, because a lot of my social security was wrapped up in being the sort of person who would do that. I would be socially scared to put it down.

My solution was very rarely to introspect on it and fight the fight directly, as I feel like is a likely takeaway from this post; that's something I could only do when the force was weak and rival forces were strong. I think a basic element involved me becoming more socially stable in other ways. I think another basic element was noticing that my overall life strategy wasn't working and was instead hurting me. I took some more hardline strategies to deal with that, (more like Odysseus tying himself to the mast than Odysseus coming to internal peace with his struggle), and then I practiced other modes of b... (read more)

7Valentine2y

I really like your contribution here. It's a great addition. Thank you. I think I see what you're pointing at. Something like… addictions can bring someone to cultivate something that (a) was very worth cultivating and (b) might have never been cultivated save for the addiction. Yes? I agree. I also think it's worth tracking why (b) happens. If you can tell something is worth cultivating, why isn't that enough? I'm guessing that part of the issue is the cultural milieu we're in (globally, not just LW). The incentives are loosely toward productivity and action. Taking the time to pay off psycho-emotional technical debt often comes with a lot of shame or inadequacy or fear. So in that environment, it makes sense to get the goods directly, even if it incurs more technical debt. One problem I'm tracking is… well, the metaphors get messy, but I'll dive ahead anyway: Too much technical debt creates a kind of memetic environment that breeds things with survival instincts, and those things like protecting their tech-debt environment. So on net, globally, I think it's actually worthwhile to let some potential Olympic athletes fail to realize their potential if it means we collectively have more psychic breathing room. And AFAICT, getting more shared breathing room is the main hope we have for addressing the real thing. (…acknowledging that Eliezer (and surely others too) explicitly disagrees with me on this point.)

[-]Ben Pace2y133

So on net, globally, I think it's actually worthwhile to let some potential Olympic athletes fail to realize their potential if it means we collectively have more psychic breathing room.
And AFAICT, getting more shared breathing room is the main hope we have for addressing the real thing.

I think this is your most general and surprising claim, and I'll hereby encourage you to write a post presenting arguments for it (ideally in a different style to the mildly pschyoactive post above, but not necessarily). I'm not sure to what extent I agree with your claim (I currently veer from 20% to 80% as I think about it) and I have some hope that if you wrote out some of the reasons that led to you believing it, it would help me make up my own mind a bit better.

[-]Valentine2y130

Here you go.

2Ben Pace2y

Very cool. I look forward to reading it.

9Valentine2y

Invitation noted. I'm open to it. I make no promises. But I like the curiosity and I'd love for what I'm seeing to land for more people and have more eyes on it.

6Ben Pace2y

Yes.

[-]Garrett Baker2y73

A summarization of the above in a way easier to evaluate would be helpful. Richard's comment does this in part, but there may be more in the post not covered by the comment.

I would usually assume a post written like this has little value to be mined, but others in comments and in upvote/downvote counts seem to disagree.

5Valentine2y

I didn't mean it to be something easy to parse and evaluate. I didn't intend it to be opaque either! But the point wasn't to make some claims that people could evaluate and think about what they agree or disagree with. The point was to resonate with something core in people who are… well, enough like I was, that they could look inside themselves and notice the path out of their misery. Breaking it down into logical statements for the cognitive mind to examine would make that much, much harder. (…which is part of how the thing I talked about in the OP works!) So if you don't want to dig into it, and you can't relate to it, don't worry about it. That just means it probably wasn't written for you.

8ShowMeTheProbability2y

From my perspective, you nailed the emotional vibe dead on. Its what I wouldve needed to hear (if I had the mental resources to process the warning properly before having a breakdown)

3Valentine2y

Good to know. Thanks for saying.

[-]Alex Flint2y62

There seems to be some real wisdom in this post but given the length and title of the post, you haven't offered much of an exit -- you've just offered a single link to a youtube channel for a trauma healer. If what you say here is true, then this is a bit like offering an alcoholic friend the sum total of one text message containing a single link to the homepage of alcoholics anonymous -- better than nothing, but not worthy of the bombastic title of this post.

5Valentine2y

If someone feels resonance with what I'm pointing out but needs more, they're welcome to comment and/or PM me to ask for more.

[-]Sphinxfire2y6-1

The truly interesting thing here is that I would agree unequivocally with you if you were talking about any other kind of 'cult of the apocalypse'.

These cults don't have to be based on religious belief in the old-fashioned sense, in fact, most cults of this kind that really took off in the 20th and 21st century are secular.

Since around the late 1800s, there has been a certain type of student that externalizes their (mostly his) unbearable pain and dread, their lack of perspective and meaning in life into 'the system', and throw themselves into the noble ca... (read more)

7bugsbycarlin2y

This has Arrested Development energy ^_^ https://pbs.twimg.com/media/FUHfiS7X0AAe-XD.jpg This is the thing to worry about. There are real negative consequences to machine learning today, sitting inside the real negative consequences of software's dominance, and we can't stop the flat fact that a life of work is going away for most people. The death cult vibe is the wild leap. It does not follow that AI is going to magically gain the power to gain the power to gain the power to kill humanity faster than we can stop disasters.

2Donald Hobson2y

There are specific technical arguments about why AI might rapidly kill everyone. You can't figure out if those arguments are true or false by analysing the "death cult vibes". Now you can take the position that death cult vibes are unhealthy and not particularly helpful. Personally I haven't actually seen a lot of death cult vibes. I have seen more "fun mental toy from philosophy land" vibes. Where total doom is discussed as if it were a pure maths problem. But if there are death cult vibes somewhere I haven't seen, those probably don't help much.

5Donald Hobson2y

This is an interesting claim. If I had a planet destroying weapon that would leave the ISS astronauts alive, would you say "don't worry about it much, it's only 3 astronaut's problem"?

2Valentine2y

I agree. I wasn't trying to speak to this part. But now that you have, I'm glad you did. I don't mean to dismiss the very real impacts that this tech is having on people's lives. That's just a different thing than what I was talking about. Not totally unrelated, but a fair bit off to the side.

[-]ShowMeTheProbability2y60

Thank you for writing this Valentine, It is an important message and I am really glad somone is saying it.
I first got engaged with the community when i was in vulnurable life circumstances, and suffered major clinical distress fixated around many of the ideas encountered here.

To be clear I am not saying rationalist culture was the cause of my distress, it was not. I am sharing my subjective experience that when you are silently screaming in internal agony, some of the ideas in this community can serve as a catalyst for a psychotic breakdown.

[-]Noah Scales2y*50

I'm a little surprised that doomerism could take off like this, dominate one's thoughts, and yet fail to create resentment and anger toward of its apparent cause ~~source~~. Is that something that was absent for you or was it not relevant to discuss here?

I wonder:

in the prediction of doom, as the threat seems to be growing closer, does that create resentment or anger at apparent sources of that doom? If I dwelled on AI existential risk, I would feel more resentment of sources of that risk.
do the responses to that doom, or desperation of measures, become

... (read more)

5Valentine2y

I'm not sure I understand your question. Do you mean, why wouldn't someone who's running the engine I describe end up resenting things like OpenAI that seem to be accelerating AI risk? For one, I think they often do. But also, it's worth noticing that the purpose of the obsession is to distract from the inner pain. Kind of like alcoholics probably aren't always upset at liquor stores for existing. And in a weird twist, alcoholics can even come to seek out relationships and situations that upset them in familiar ways. Why? Because they know how to control that upset with alcohol, which means they can use the external upset as a trigger to numb out instead of waiting for glimmers of the internal pain to show up inside them. Not all addiction designs do this. But it's a common enough pattern output to be worth acknowledging. I'm not sure if that's what you were asking about though.

3Noah Scales2y

You wrote: Oh. Good! I'm a bit relieved to read that. Yes, that was the fundamental question that I had. I think that shows common-sense. I'm curious what you think a sober response to AGI research is for someone whose daily job is working on AI Safety, if you want to discuss that in more detail. Otherwise, thank you for your answer.

3Valentine2y

Quite welcome. I'm not really up for surmising about this right now. It's too tactical. I think the clarity about what to do arises as the VR goggles come off and the body-level withdrawal wears off. If I knew what it made sense for people to do after that point, we wouldn't need their agentic nodes in the distributed computation network. We'd just be using them for more processing power. If that makes sense. I bet I could come up with some general guesses. But that feels more like a musing conversation to have in a different context.

[-]Lao Mein2y51

How bad are things, really? I'm not part of EA/Rat/AI risk IRL, so I dont have first-hand experience. Are people actually having mental breakdowns over the control problem? Some of the comments here seem to imply that people are actually experiencing depersonalization and anxiety so bad it's affecting their work performance specifically because of AI concerns. And not just them, but multiple people they work with. Is the culture at AI alignment orgs really that bad?

5Valentine2y

I'm not up to date on this. I've been out of the community since 2018. But back when co-running CFAR there absolutely was an issue with people going psychotic or having panic attacks or crushing depression in the face of this stuff. Absolutely. (…with caveats around how discerning causation vs. correlation is very tricky, there are selection effects, etc. But it was a clear enough impression that CFAR staff had explicit strategies in case someone started showing signs of having a manic episode. As just one small sign of many.)

3Lao Mein2y

How did we get here? How did Rationality go from being about winning at life to a Cthullu RPG? I'm pretty interested in AI, and the fact that stuff like this is real (it still doesn't feel real, it feels like LARP) has really turned me off of alignment research. On a personal level, the literally-having-coworkers-get-driven-insane of alignment research seems a lot worse than merely-slightly-increasing-existential-risk of capabilities research.

[-]lc2y1820

I'd rather go insane than "slightly increase" existential risk.

[-]Valentine2y100

I think this is maybe implied by Aiyen's comment, but to highlight an element here:

This way of thinking doesn't have you trade sanity for slight xrisk decrease.

It has you trading sanity for perceived slight xrisk decreases.

If you start making those trades, over time your ability to perceive what's real decays.

If there's anything that has anything whatsoever like agency and also benefits from you feeding your sanity into a system like this, it'll exploit you right into a Hell of your own making.

Any time you're caught in a situation that asks for you to make a trade like this, it's worth asking if you've somehow slipped into the thrall of an influence like this.

It's way more common than you might think at first.

3Aiyen2y

On the one hand, that's literally true. On the other, I feel like the connotations are dangerous. Existential risk is one of the worst possible things, and nearly anything is better than slightly increasing it. However, we should be careful that that mindset doesn't lead us into Pascal's Muggings and/or burnout. We certainly aren't likely to be able to fight existential risk if it drives us insane! I strongly suspect that it's not self-sacrificing researchers who will solve alignment and bring us safely through the current crisis, but ones who are able to address the situation calmly and without freaking out, even though freaking out seems potentially justified.

8Valentine2y

It's woven deep in LW's memetic history. Eliezer created LW in order to sort of spell out his understanding of the Art. Part of the point of that was to act as a kind of combo summon spell and filter for potential AI risk researchers for MIRI. The reason was in the wake of Eliezer noticing he had been ludicrously wrong in his thinking about AI, and noticing how hard it is to think about AI clearly, and how AI researchers in particular couldn't think clearly about the topic clearly. So the "This is literally unfathomably important, we have to actually solve this thinking clearly thing or literally everything anyone could ever value is going to get permanently destroyed!" tone was woven in from the very beginning. This is just one example. So, yeah. It's always been like this. Just varying degrees of cloaked vs. transparent, with corresponding peripheral folk sometimes floating in who aren't as much about AI risk. But the thrust as a whole has always come from this Lovecraftian core.

5janus2y

turns out life is a Cthullu RPG, so we gotta win at that

1ztzuliios2y

It was always a Cthulhu LARP. Remember that one thing? Groups polarize over time. One of the ways to signal group membership is to react explosively to the things you're supposed to react explosively to. This is why as politics in the US have polarized, everyone has grown more breathless and everything is ${OUTGROUP}ist. You gain ${KARMA} by being more breathless and emotional. You can only stop this with explicit effort, but if you do that, you look dissociated, disembodied, autistic, and the social pressure against that is stronger than the social pressure against letting (totally normal, status-quo) group polarization get to the point of literally mindkilling people. The military also has techniques to avoid this, but there are similar social pressures against them from both sides of the equation, because the military is low-status both to the postrationalist types (like OP) and to actual rationalists (like the people setting social norms at CFAR). So you don't see those being used. It's completely possible to have the alignment research you want, you'll just have to set the group norms, which is a completely orthogonal problem that involves a lot of things besides alignment research. Personally I think this would be a very good thing to do and I encourage you to try for the experience of it if nothing else.

0Slider2y

This is a real stigma consideration. Reminder that water is water. Autism as something inherently negative is ablism and improper.

4ztzuliios2y

I completely agree and I think that levying the charges "disembodied" against anything on the opposite side of the mental dichotomy of "woo" is a weasel-word for the ableist slur of "autistic." I'm sorry this wasn't more clear, but I thought that sentence was fairly dripping as is. I've written about this before as it applies to this topic, which is not to excuse the harm I've done if I've triggered you, but to show that I've precommitted to this stance on this issue.

3Slider2y

I feel slightly nervous I am overtly spending internet ink on this but here I go. I was unsure how obvilously wrong it was ment to appear. To me saying that "someone seems cold" is not problematic. "dissociated" and "disembodied" read to me to be part of natural feeling (expression), somebody could mean technical things with them and not have a attitude loaded into them. Those parts did not constitute drippingness for me. For autistic there was no technical meaning that could make sense. I was not triggered but it did cross my mind that not everyone thinks that is unbased take to me (and kind of categorising this knowledge as common only in the small subcommunity). Having those fly without anybody batting an eyelid would be normalising the hateful conduct. I was unsure whether the eyelid was batted already so I batted a separate eyelid. And tried to include indicators that it is a mild reaction (a kind of messaging I have reason to believe I frequently screw up). I do reflect that if I was in the context where this could be assumed common knowledge I would not probably be making this move (bat the eyelid). So I am wondering whether it connects to the object level phenomena (of groups polarising) where people harden their signals so that outsiders with more noise and less decoding ability do not get unintended messages.

3ztzuliios2y

Lots of ink, but lots to think about. I'm thankful for this post fwiw. The "no technical meaning" could maybe be an indicator of sarcasm. But you're right that there was no way for you to know I wasn't just misapplying the term in the same way as the OP. I don't think this relates to group polarization per se but I take your point. I didn't mean "triggered" to mean extremely so, someone can be mildly triggered and again, I apologize for (in my perception, based on your comment) doing that. I think you did the right thing.

2Slider2y

With no resonable way of knowing without context I am using "technical" here in a very idiosyncratic way. If two speech acts that have very different connotations and then strip them of the connotations if they are the same then the technical meaning is the same. If someone is being hateful I often proceed to "fix the message from them" mentally in my receiving end. So while I starkly reject parts of it, rejecting everything of it punishes also the non-hateful parts. Thus I have the cognitive task of "what they should have said". If there is no innocent message left after removing the hate, it is pure hate. This is a kind of "could a reasonable opiner opine this?" standard. It is easy to read "disembodied" in a ablist way but it might just be a clumsy way to refer to low charisma (is is "repairable"). So after phrasing incompetence is exhausted an assumption of malice starts. To have the statistical mean human deduce "That guy gets passionate in an unnatural way -> that guy is autistiic" has low plausibility. Backtracing where this logic would be natural, worrying about upholding a mask about a behaviour that has lots of details and has high fluency from the mimic target making it highly likely to be a statistical outlier that a masking strategy does not cover well (this is not meant to be a mask review). Confusion, "stiffness" or "odd feeling" would represent what happens in situations like these. Zero to 100% autistic label is irrealistic. The average hater is not that informed.

[-]Shmi2y50

I agree with everything, though this is just a very long way to say the https://en.wikipedia.org/wiki/Serenity_Prayer

4Valentine2y

I think I'm saying something quite a bit more precise than the Serenity Prayer. But I agree, it's related.

[-]Ape in the coat2y51

I think you are pointing at an important referent. There are probably a lot of people who will benefit from reading this post and thus I'm glad that you wrote it. That said, you appear to have written it in a deliberately confusing manner. You probably have your reasons. Maybe you believe that this way would be better for the people you are trying to help. I'm not an expert in Lacanianism, but I think this is wrong, both ethically and epistemologically. The are also a lot of people who will misunderstand this post incorrectly and for whom reading it will c... (read more)

[-]Ansel2y52

Strongly upvoted, I think that the point about emotionally charged memeplexes distorting your view of the world is very valuable.

[-]Double1y40Review for 2022 Review

Ideally reviews would be done by people who read the posts last year, so they could reflect on how their thinking and actions changed. Unfortunately, I only discovered this post today, so I lack that perspective.

Posts relating to the psychology and mental well being of LessWrongers are welcome and I feel like I take a nugget of wisdom from each one (but always fail to import the entirety of the wisdom the author is trying to convey.)

The nugget from "Here's the exit" that I wish I had read a year ago is "If your body's emergency mobilization sys... (read more)

3Valentine1y

I like the tone of this review. That might be because it scans as positive about something I wrote! :D But I think it's at least in part because it feels clear, even where it's gesturing at points of improvement or further work. I imagine I'd enjoy more reviews written in this style. If folk can find ways of isolating testable claims from this post and testing them, I'm totally for that project. The claim you name isn't quite the right one though. I'm not saying that people being stressed will make them bad at AI research inherently. I'm saying that people being in delusion will make what they do at best irrelevant for solving the actual problem, on net. And that for structural reasons, one of the signs of delusion is having significant recurring sympathetic nervous system (SNS) activation in response to something that has nothing to do with immediate physical action. The SNS part is easy to measure. Galvanic skin response, heart rate, blood pressure, pupil dilation… basically hooking them up to a lie detector. But you can just buy a GSR meter and mess with it. I'm not at all sure how to address the questions of (a) identifying when something is unrelated to immediate physical action, especially given the daughter's arm phenomenon; or (b) whether someone's actions on net have a positive effect on solving the AI problem. E.g., it now looks plausible that Eliezer's net effect was to accelerate AI timelines while scaring people. I'm not saying that is his net effect! But I'm noting that AFAIK we don't know it isn't. I think it would be extremely valuable to have some way of measuring the overall direction of some AI effort, even in retrospect. Independent of this post! But I've got nuthin'. Which is what I think everyone else has too. I'd love for someone to prove me wrong here. This is a beautiful idea. At least to me.

1Double1y

I'm glad you enjoyed my review! Real credit for the style goes to whoever wrote the blurb that pops up when reviewing posts; I structured my review off of that. When it comes to "some way of measuring the overall direction of some [AI] effort," conditional prediction markets could help. "Given I do X/Y, will Z happen?" Perhaps some people need to run a "Given I take a vacation, will AI kill everyone?" market in order to let themselves take a break. What would be the next step to creating a LessWrong Mental Health book?

0[comment deleted]1y

[-]alkexr2y42

I immediately recognize the pattern that's being playing out in this post and in the comments. I've seen it so many times, in so many forms.

Some people know the "game" and the "not-game", because they learned the lesson the hard way. They nod along, because to them it's obvious.

Some people only know the "game". They think the argument is about "game" vs "game-but-with-some-quirks", and object because those quirks don't seem important.

Some people only know the "not-game". They think the argument is about "not-game" vs "not-game-but-with-some-quirks", and ob... (read more)

[-]Coafos2y4-2

Thank you for writing this post, I think this is a useful framing of this problem. For me personally, the doom game is fun, imho I have more motivation to do things and I become more self-confident. (if it ends what worse could happen) But that's for me, with my socially isolated Math/ComSci/CosHo background.

For others, I don't think it's a good game. I kinda noticed the tons of psychotic breakdowns around the field and, like, that's bad, but I could not have articulated why it was bad.

And even for me, I might kinda overshoot with the whole information hazard share-or-not thinking. It's better if you're in charge of the game and not let the doom game play you.

1Valentine2y

Awesome. I have deep respect for this kind of conscious game-playing. Rock on!

[-]Nicholas / Heather Kross2y32

Strong agree with TekhneMakre's comment.

Purely on Valentine's own professed standards: Of all the ways to "snap someone out of it", why pick one that seems the most like brainwashing? If the FBI needs to un-brainwash a dangerous cult member, do they gaslight them? Do they do a paternalistic "if you feel angry, that means I'm right" maneuver? Do they say "well I'm not too concerned if you think I'm right" to the patient?

(Also... FWIW, the most doomsday-cultish emotionally-fraught posts I've seen in the rationality community are, by percentage of posts, most... (read more)

-14Valentine2y

[-]Peter Hroššo2y30

I very much agree that actions motivated by fear tend to have bad outcomes. Fear has subtle influence (especially if unconscious) on what types of thoughts we have and as a consequence, to what kinds of solutions we eventually arrive.

And I second the observation that many people working on AI risk seem to me motivated by fear. I also see many AI risk researchers, who are grounded, playful, and work on AI safety not because they think they have to, but because they simply believe it's the best thing they can do. I wish there would be more of the latter, bec... (read more)

3Valentine2y

Agreed. I don't know many, but it totally happens. Yep. I've had to just fully give up on not being polarizing. I spent years seriously trying to shift my presentation style to be more palatable… and the net effect was it became nearly impossible for me to say anything clearly at all (beyond stuff like "Could you please pass the salt?"). So I think I just am polarizing. It'd be nice to develop more skill in being audible to sensitive systems, but trying to do so seems to just flat-out not work for me. Alas. Thank you for saying so. My pleasure.

[-]Michael Roe1y*21

I think this post is a good one, even though personally I'm not hung up on AI doom; I think this area of research is cool and interesting, which is a rather different emotion from fear.

My immediate thought is that Cognitive Behevioral Therapy concepts might relevant here, as it sounds like a member of the family of anxiety disorders that CBT is designed to treat.

And also, given this a group phenomenon rather than a purely individual one, there's something of the apocalyptic religious cult dynamic going on.

one thing that can be kind of irritating about CBT ... (read more)

[-]Yoav Ravid2y21

If your body's emergency mobilization systems are running in response to an issue, but your survival doesn't actually depend on actions on a timescale of minutes, then you are not perceiving reality accurately.

You are locked in a room. You are going die of thirst in a few days. The door has a combination lock. You know the password is 5 digits (0 to 9). If it takes you one second to try each combination, it's going to take you 27.7 hours to try all the combinations (so half that on average to find the right one). Your survival doesn't depend on your action... (read more)

[-]swarriner2y155

Unless I'm very much mistaken, emergency mobilization systems refers to autonomic responses like a pounding heartbeat, heightened subjective senses, and other types of physical arousal; i.e. the things your body does when you believe someone or something is coming to kill you with spear or claw. Literal fight or flight stuff.

In both examples you give there is true danger, but your felt bodily sense doesn't meaningfully correspond to it; you can't escape or find the bomb by being ready for an immediate physical threat. This is the error being referred to. In both cases the preferred state of mind is resolute problem-solving and inability to register a felt sense of panic will likely reduce your ability to get to such a state.

2Noosphere892y

This. I think a lot of the problems re emergency mobilization systems relate to that feeling of immediateness, when it's not. I think a lot of emergencies are way too long-term for us, so we apply emergency mobilization systems even when they aren't there.

[-]Heron1y1-9Review for 2022 Review

This short post is astounding because it succinctly describes, and prescribes, how to pay attention, to become grounded when a smart and sensitive human could end up engulfed in doom. The post is insightful and helpful to any of us in search of clarity and coping.

[-]David Bahry2y1-10

(…although I doubt he consciously intended it that way!)

I'm pretty sure Eliezer's "Death With Dignity" post was an April Fool's joke.

[-][anonymous]2y1-4

This is the basic core of addiction. Addictions are when there's an intolerable sensation but you find a way to bear its presence without addressing its cause. The more that distraction becomes a habit, the more that's the thing you automatically turn to when the sensation arises. This dynamic becomes desperate and life-destroying to the extent that it triggers a red queen race.

I doubt that addiction requires some intolerable sensation that you need to drown out. I'm pretty confident its mostly a habits/feedback loops and sometimes physical dependence.

1[anonymous]2y

For instance, ~1 billion people worldwide are addicted to caffeine. I think that's just what happens when a person regularly consumes coffee. It has nothing to do with some intolerable sensation.

2Valentine2y

I'm guessing we're using the word "addiction" differently. I don't deny that there's a biological adaptation going on. Caffeine inhibits adenosine, prompting the body to grow more adenosine receptors. And stopping caffeine intake means the adenosine hit is more intense & it takes a while for the body to decide to break down some of those extra receptors. (Or something like that; I'm drudging up memories of things I was told years ago about the biochemistry of caffeine.) But here's the thing: Why does that prompt someone to reach for coffee? "It's a habit" doesn't cut it. If the person decides to stop caffeine intake and gets it out of their house, they might find themselves rationalizing a visit to Starbucks. There's an intelligent process that is agentically aiming for something here. There's nothing wrong with feeling tired, sluggish, etc. You have to make it wrong. Make it mean something — like "Oh no, I won't be productive if I don't fix this!" This is the "intolerable" part. Intolerability isn't intrinsic to a sensation. It's actually about how we relate to a sensation. I've gone through caffeine withdrawal several times. Drudged through the feelings of depression, lethargy, inadequacy, etc. But with the tone of facing them. Really feeling them. It takes me just three days to biologically adapt to caffeine, so I've done this quite a few times now. But I actually dissolved the temptation to stay hooked. Now I just use caffeine very occasionally, and if it becomes important to do for a few days in a row… I just go through the withdrawal right afterwards. It's not a big deal. Which is to say, I've dissolved the addiction, even though I can still biologically adapt to it just like anyone else. I would say I'm not addicted to it even when I do get into an adaptive state with it. Does that clarify what I'm talking about for you?

1[anonymous]2y

It does clarify what you are talking about, thank you. Now it's your use of "intolerable" that I don't like. I think most people could kick a coffee addiction if they were given enough incentive, so withdrawal is not strictly intolerable. If every feeling that people take actions to avoid is "intolerable", then the word loses a lot of its meaning. I think "unpleasant" is a better word. (Also, the reason people get addicted to caffeine in the first place isn't the withdrawal, but more that it alleviates tiredness, which is even less "intolerable.") Your phrasing in the below section read to me like addiction is symptomatic of some character defect. If we replace the "intolerable" with "unpleasant" here, it's less dramatic and makes a lot more sense to me. I don't think this matters much for the rest of the post. It just felt like this mischaracterizes what addiction is really about.

[-]Iknownothing1y00

I think calling things a 'game' makes sense to lesswrongers, but just seems unserious to non lesswrongers.

[-]S Benfield1y-4-1

Look, I can go into mania like anyone else here probably can. My theories say that can't be genius level without it and that it comes with emotional sensitivity as well. Of course if you don't believe you have empathy, you won't, but you still have it.

I am not an AI doom and gloomiest. I adhere to Gödel, to Heisenberg, and to Georgeff. And since we haven't solved the emotional / experience part of AI, there is no way it can compete with humans creatively, period. Faster, yes. Better, no. Objectively better. not at all.

However, if my theory of t... (read more)

[+]Decius2y-90

[+]Valentine1y-16-14Review for 2022 Review

Moderation Log