Thanks for this comment. I was both very moved by this post and unwilling to lean into it due to fears I couldn't articulate. "fear of being eaten" is a pretty good match for what I was feeling, and having read the post I feel much more able to distinguish shadowmoth situations from being-eaten situations.
Some aspects that seem important to me for distinguishing between the two:
- do you actually want the goal that struggling is supposed to bring you closer to? Did you choose it, or was it assigned to you? how useful is the goal to the non-helper?
- how much work is the non-helper saving themselves by refusing to help you? In the shadowmoth case it was hurting the non-helper, which makes not-helping more likely to be genuinely for the moth's benefit.
- is the struggling recurring indefinitely, or is there some definitive endpoint? or at least, a gradual graduation to a higher class of problem?
- can you feel something strengthening as you struggle?
- is there reason to believe you'll eventually be capable of doing the thing?
- is this goal the best use of your limited energy? maybe someone should help you out of this cocoon so you can struggle against a more important one.
- are you going into freeze response?
There's also the question of the non-helper as an instance of a class, and you as an instance of a class, and the resulting implied ecology. Or to say it a different way: apply TDT to the shadowmoth / meal question. To say it a third way: if people like me react to situations like this--involving some relationship with someone or something--in such-and-such a way, then what trophic niche are we opening up, i.e. what sort of food are we making available for what sort of predator?
1I found this a particularly helpful lens.
I had had individual thoughts like "of course in this case the alien woman is, you know, bad for kidnapping him and torturing him", but the particular ecosystem frame feels probably-useful for generating followup questions. (It's also thematically resonant with the themes in the book!)
This thread did motivate me to add an additional disclaimer to the post.
2I was a miserable child. When I was nine years old I remember watching one and thinking "I have almost a decade left to serve. This is a long sentence for an adult and im just a kid. But at least I will get out one day".
I was eventually set free. But until my freedom came all I could really do was bide my time and try to cope with the torture. And I most certainly consider it torture in retrospect. I was physically assaulted by my dad and I was horribly, horribly sleep deprived. But I managed to keep some of my sanity and pick up some MTG cards I later sold at a large profit. It could have been a lot worse for future me.
A thing I am interested in but can't tell from this comment is whether, as that kid, reading this post would have been helpful or harmful (I'd guess harmful, but not overwhelmingly)
I greatly enjoyed this book back in the day, but the whole scenario was wild enough to summon the moral immune system. Past a certain point, for me it’s a safe default to put up mental barriers and actively try not to learn moral lessons from horror fiction. Worm, Gideon the 9th, anything by Stephen King- great, but I don’t quite expect to learn great lessons.
While rejecting them as sources of wisdom now, I can remember these books and return to them if I suddenly need to make moral choices in a world where people can grow wiser by being tortured for months, or stronger by
killing and then mentally fusing with your childhood friend. or achieve coordination by mind controlling your entire community and spending their lives like pawns
Though compare and contrast Dune's test of the gom jabbar:
You've heard of animals chewing off a leg to escape a trap? There's an animal kind of trick. A human would remain in the trap, endure the pain, feigning death that he might kill the trapper and remove a threat to his kind.
Even if you are being eaten, it may be right to endure it so that you have an opportunity to do more damage later.
I mean, we're getting this metaphor off its rails pretty fast, but to derail it a bit more:
The kind of people who lay human-catching bear traps aren't going to be fooled by "Oh he's not moving it's probably fine".
Everybody likes to imagine they'd be the one to survive the raiding/pillaging/mugging, but the nature of these predatory interactions is that the people doing the victimizing have a lot more experience and resources than the people being victimized. (Same reason lots of criminals get caught by the police.)
If you're being "eaten", don't try to get clever. Fight back, get loud, get nasty, and never follow the attacker to a second location.
I clicked the link hoping to expand the metaphor into some object level example that applied in this case but instead just found more metaphor (at least, I hope what I found was more metaphor) :P
Here's a non-metaphor!
“Cryptobenthics do one thing particularly well: getting eaten.”
I guess the moral of that is, don't be a cryptobenthic.
1At the beginning of the post, I wrote: [update: I deleted it because I didn't really expect it to help people and undercut the opening]
This post is probably hazardous for one type of person in one particular growth stage, and necessary for people in a different growth stage, and I don't really know how to tell the difference in advance.
If you read it and feel like it kinda wrecked you send me a DM. I'll try to help bandage it.
I thought about putting it again at the end, where people would actually be more likely to see it when they realized they needed it. But, also, it felt like it would undercut the poetic heft of the end of the post. But then I realized I could put it in the comments here, which feels like a reasonable middle ground.
1I am but puzzle about how your post could wreck someone, it's a great piece to help those with fears about something they cannot control to not worry about it too much or understanding the importance of learning to grieve. If I were in those spots, the message is an optimistic one
There is an optimistic message there, yes - equanimity and internal locus of control and so on are valuable and useful skills. But if you're not ready to learn them, and you're not in a place where you are prepared to walk the path to truly internalize them, they can feel impossible, and make you feel inadequate and incompetent, and demoralize you, and lead to depression, or dissociation. Those things may also happen if you are prepared, but the difference is whether you're able to get through to the other side.
I am
What's "I am" mean in that sentence?
I didn't have an exact model of what I'd expect to go wrong, but was generally aware that it might pack a big emotional wollop for people in particular spaces, (two people have messaged me so far to say they were shaken by it, in a good way), and it seemed generally better to say "hey this gun is loaded" and be ready to deal with whatever fallout there turned out to be.
But, at the very least, people might just have a lot of stuff going on and don't necessarily have time to have some hopefully-cathartic-but-meanwhile-destabilizing-or-intense-reorientation..
...
[edit: Maybe one particular thing I'd be worried about are people with a belief that things-will-be-okay-soon-because-something-will-end-or-someone-will-help them, and then this post questions that belief, and suddenly a psychologically loadbearing belief is gone and they don't immediately have the tools to deal with it]
I am but puzzle about how your post could wreck someone
when someone does not have the capabilities to face those fears. Even just meditating is dangerous for some people because it makes them face something they are not equipped to face. In order to learn, one must face challenges with the right level of difficulty for them at that point in time. Too easy and there's no learning; too difficult and it is a wall instead of a challenge. If the challenges are psychological or similar, this wall may be something that hurts, only hurts, with no -or minimal- gains.
I think it's worth pointing out that the ultimate outcome here is "Jacen becomes a sith lord and has to be put down by Luke and his sister Jaina to save the New Republic/Galactic Alliance."
In order words, in the Star Wars universe, as usual, technology never works all that well for solving real problems, philosophy is dangerous for anyone who questions received wisdom, and human minds can't handle the complexity of reality or of using any non-Jedi philosophy for relating to The Force. Even an avowed pacifist will fall if they stray from the received path, even based on correct understanding and insight, even if it's only to learn other traditions that are already out there. The Light and Dark sides are closer to order/chaos than good/evil, change is chaos, and chaos corrupts, even when the change is otherwise good.
In other words, Jacen has more humility in the "I still have a lot to learn, I'm not a hero" sense, more growth mindset, and more tsuyoku naritai than possibly any other Jedi in the EU, and it destroys him.
To be clear, I think your subsequent analysis is good and insightful. Just, be very careful using anything from Star Wars as a guidepost, because more often than not the lesson is "Problems are caused by not following the established traditions, and even the plucky rebels are mostly only good when they're trying to restore what already once was, so don't reach for much more than you're granted."
It is unreasonable to expect an output from biological evolution to do something reasonable in a situation vastly different from the situations it evolved in. In this case, you aren't going to learn much from the state of mind of a human who has been tortured more than his ancestors could have been tortured.
Trying to extract guidance from that story seems like an example of generalizing from fiction.
I'm not sure how the rest of the article is connected to the fiction at the beginning. Given that people tend to generalize from fiction and get to false conclusions, I'm uncomfortable with taking it seriously.
1Certainly seems a reasonable worry.
My angle here is "this obviously doesn't tell you anything about what humans can or should do when they are being maximally tortured. But it is inspirational the way stories can often be in a way that is more about making something feel like a visceral possibility, which didn't previously feel like a visceral possibility."
And then, the concrete details that follow are true (well, the metastrategy one is "true" in that "this is why I'm doing it this way", it doesn't really go into "but how well does it work actually?".
The thing I would encourage you to do is at least consider, in various difficult circumstances, whether you can actually just shut up and do the impossible, and imagine what it'd look like to succeed. And then concretely visualize the impossible-seeming plan and whatever your best alternative is, and decide between them as best you can.
This may be an example, but I don't think it's an especially central one, for a few reasons:
1. The linked essay discusses, quite narrowly, the act of making predictions about artificial intelligence/the Actual Future based on the contents of science fiction stories that make (more-or-less) concrete predictions on those topics, thus smuggling in a series of warrants that poison the reasoning process from that point onward. This post, by contrast, is about feelings.
2. The process for reasoning about one's, say, existential disposition, is independent of the process for reasoning regarding the technical details of AI doom. Respective solutions-spaces for the question "How do I deal with this present-tense emotional experience?" and "How do I deal with this future-tense socio-technical possibility?" are quite different. While they may feed into each other (in the case, for instance, of someone who's decided they must self-soothe and level out before addressing the technical problem that's staring them down or, conversely, someone who's decided the most effective anxiety treatment is direct material action regarding the object of anxiety), they're otherwise quite independent. It's useful to use a somewhat different (part of your)self to read the Star Wars Extended Universe than you would use to read, i.e., recent AI Safety papers.
3. One principle use of fiction is to open a window into aspects of experience that the reader might not otherwise access. Most directly, fiction can help to empathize with people who are very different from you, or help you come to grips with the fact that other people in fact exist at all. It can also show you things you might not otherwise see, and impart tools for seeing in new and exciting ways. I think reading The Logical Fallacy of Generalization from Fictional Evidence as totally invalidating insights from fiction is a mistake, particularly because the work itself closes with a quote from a work of fiction (which I take as pretty strong evidence the author would not endorse using the work in this way). If you don't think your implied reading of Yudkowsky here would actually preculde deriving any insight whatsoever from fiction, I'd like to hear what insights from fiction it would permit, since it seems to me like Ray's committing the most innocent class of this sin, were it a sin. It's possible you just don't think fiction is useful at all, and in that case I just wouldn't try to convince you further.
4. I read Ray's inclusion of the story as immaterial to his point (this essay is, not-so-secretly, about his own emotional development, with some speculation about the broader utility for others in the community undergoing similar processes). It's common practice in personal essay writing to open with a bit of fiction, or a poem, or something else that illustrates a point before getting more into the meat of it. Ray happens to have a cute/nerdy memory from his childhood that he connects to a class of thinking that in fact has a rich tradition (or, multiple rich traditions, with parallel schools and approaches in ~every major religious lineage).
[there's a joke here, too, and I hope you'll read my tone generously, because I do mean it lightheartedly, about "The Logical Fallacy of Generalization from The Logical Fallacy of Generalization from Fictional Evidence"]
Curated. I think there is something of a rationalist virtue of maintaining agency and a locus of control in whatever situation you find yourself in, that this post points to. I think that I've not quite named it right, of course in reality you are often not in control, but maintaining agency such that you can direct reality the moment that it's possible to, is pretty key.
I can't remember whether it's Jacen, or the woman, who goes on to say "you might sit with it, and let it know that it's not alone, and do what you can to comfort it."
"'And also, perhaps,' [the woman] went on, 'you might stop by from time to time, to let the struggling, desperate, suffering creature know that it is not alone. That someone cares. That its pain is in the service of its destiny.'"
It's the woman.
And also I guess I'd like to register another weird coincidence, Traitor was also one of my favourite stories growing up. I disagree with you about the second half of the book being forgettable though. Ganner Rhysode also felt important.
1You're right about growing. Typically, people on a 'growth agenda' search for and adopt new ways/ideas over time. These 'methods' are initially freeing us to grow, but ultimately the method needs to be jettisoned as it becomes a trap for habitual or ritualised behaviour. The self-actualised state is achievable, but not sustainable because the shifts in world around us and within us. Helping or serving others seems obligatory to me.
Loved the essay; had a surprisingly negative emotional response to seeing what feels like unnecessary cruelty to poetry at the end though!
The act of taking someone else's poem, changing a mere three words of it (in a way that is both unnecessary and aesthetically detrimental) and then crediting oneself as the author feels icky.
Would the unbastardised verse from Invictus by Henley himself not have worked here?
"It matters not how strait the gate, How charged with punishments the scroll, I am the master of my fate I am the captain of my soul."
I found Kenzie's version of it moving specifically because of the deliberate difference and wouldn't have felt motivated to include the original at all. I do realize it doesn't actually change the semantic content, but it felt surprisingly poetic on it's own and connected it more clearly to other ideas I cared about.
But, I wasn't at all trying to deny Henley credit and have added him there.
I would want to remind any reader of poetry that vast swaths of poetry lift entire lines, structures, and meters from other poems. This is not a new thing to be done in the space of poetry and I found that this version of the poem respects the original by allowing the careful reader (such as yourself) to look up the original and compare the two.
Too late "not to torture people to make them grow their minds" 💁
It is what school education is about, we are forcing insane amounts of ideas and data through children's minds. That causes radical and powerful transformations in their monkey brains.
That's not very pleasant for most kids, learning. Some teenagers develop obvious disturbances in social and overall behavior (we call them "geeks", "girls-bas-bleau")
But as result some of the "tortured souls" turn into readers and authors of this forum :-)
One of my favorite stories growing up was Star Wars: Traitor, by Matthew Stover.
The book is short, if you want to read it. Spoilers follow. (I took a look at it again recently and I think it didn't obviously hold up as real adult fiction, although quite good if you haven't yet had your mind blown that many times)
One anecdote from the story has stayed with me and shows up in my worldview from time to time.
The story begins with "Jacen Solo has been captured, and is being tortured."
He is being extremely tortured. He's in a white room, with a device connected throughout his nervous system, inflicting maximum white-hot agony. Whenever the suffering becomes so great that his body tunes it out, it gives him just enough time to relax to hurt all the more when it abruptly starts up again.
Every day, an alien woman comes to visit him. She talks to him, during lulls in the torture, about philosophy and life and history. She asks him what he thinks about things. He says "I think I'm being fucking tortured. Can you please help?", and she kind of ignores him and goes on talking about philosophy and life and history and stuff. Sometimes he tries more clever ways to pry information out of her, to no avail.
After several weeks (or months?) of this, at some point he asks again "Seriously what the hell? Who are you? What is happening to me? Why are you torturing me, or, not helping? What is going on?"
She says: "Do you know what a shadowmoth is?"
"Answer my fucking question"
"Do you know what a shadowmoth is?"
"Fine, a shadowmoth is a life form native to Coruscant. They're silicon-based, their wingflutes make music once they leave the cocoon and can fly."
She nods, and says "Once, I saw a shadowmoth in its cocoon, struggling. Like you, I am sensitive to the force – I didn't merely empathize with it, but I felt it. When shadowmoths strive to break free from their cocoon, they are in pain. I felt it's pain. And I felt it's envy of the other moths, already free, flying, making music. I wanted to help it."
Jacen shudders. "Oh no, you didn't –"
"Yes. I cut it free."
"No –"
"It sounds like you know what happened next?"
Jacen says: "If they don't escape from the cocoon on their own, they'll never be strong enough to fly, and they suffer and starve to death, worse than what they were experiencing in the cocoon."
"Indeed."[1]
The two of them sit for awhile. The torture machine gives Jacen another wave of White Pain. Then the woman says "Suppose you did want to help a shadowmoth. What would you do, if you couldn't cut it free? You might sit with it, and let it know that it's not alone, and that it's pain is in the service of it's destiny."
And then she leaves for the day.
And, then, it occurs to Jacen:
It's been months. They haven't asked him any questions. There is no indication that they particularly want anything from him. There is no indication that the White Hot Torture will ever stop.
What would he do, if this was literally the only thing there would ever be? Just him, and the White, and this inscrutable alien woman...
...basically, the only action he can do, is to change his relational stance to the torture, and learn to stop caring about it.
The last sentence of that chapter is "Jacen Solo begins to eat away at the White."
...
A few weeks later, he has basically invented Buddhist Enlightenment from first principles. He feels the pain, but does not suffer. He converses freely with the woman about philosophy and history and life and stuff.
And, then, his captors say "Congratulations. You have passed the first test. You are now ready for the second phase of your training."[2]
Now, there are a few things I have to say about this.
First, to forestall the obvious objections: The transhumanist answer to "what if painful suffering is necessary to learn valuable life lessons?" is "skill issue."
If you can't figure out how to let shadowmoths out of cocoons without them suffering and dying, maybe you can just invent better biotech until this is no longer necessary. In meantime, sure, have some equanimity about it. But don't let that equanimity stop you from trying to improve the world.
But, better biotech might take thousands of years (or, like, at least a decade) to invent, and you might have to deal with a situation in the meanwhile anyway. And I do think there is something powerful here about never losing your locus of control. And there’s something important about "Sometimes, at least for now, the best way to learn some things involves struggling and figuring things out for yourself. Teachers can help point you to the door but you'll have to walk through it yourself."
Also, perhaps the even more obvious first objection: in this story, the woman is out to manipulate Jacen into becoming a different sort of person. In the context of the story, I think that many of the changes to Jacen's character are actually good by Jacen's original lights[3], but to be clear I do not endorse kidnapping people and torturing them even if you can argue it's for their own good somehow. Or, most things that that would directly be a metaphor for[4].
But in the context of "things you actually consented to", or "things reality is forcing upon you without anyone else really being involved at all", there are two particular angles that come up for me a fair amount.
The first angle is metastrategy
The goal of my rationality training investigation is not to produce people who are good at researching a particular domain, or solving a particular class of problem. If you are working in a known field, the thing to do is find experts in that field and learn from them. Sometimes they may need to improve their pedagogy, or learn to articulate their tacit knowledge better. You can learn to extract tacit knowledge from people, and you can learn to extract more knowledge from textbooks, etc.
But, for problems we don't understand yet, there may just not be an established field to learn from.
There, you need the skill of either inventing new skills, or connecting dots between disparate existing fields that nobody has yet integrated. I think you need some kind of generalized research taste, that tells you what to do when nobody in the world is capable of authoritatively giving you the answer, and the specialized research tastes of various fields might be subtly leading you astray, like a physicist assuming they can model everything as spherical cows, or an ML researcher assuming they can continue to be behaviorist about their research as they push the frontier towards superintelligence.
For this reason, my training is very "shadowmoth-y". Much of my process is "make people solve a problem they don't know how to solve, and challenge them to figure it out."
I think there is some art to being "exactly the right amount of cryptic and annoying", and to finding problems that form a reasonable difficulty curve for people with various different starting skills. I don't know that I've hit this balance well yet. The transhumanist answer to shadowmothism still applies – if I can give people clues that help them learn the right metaskills without being so painful and annoying, all the better.
But I try, where possible, to only give people the earliest hints on a thought chain, nudging them towards the right metatools to invent/remember the right metatools to find/remember the right object level tools to solve a given problem.[5]
The second angle is something like "resilient self-ownership and locus of control."
Long ago, a friend of mine had recently emotionally grokked existential risk – he had known about and "believed in" it for years, but somehow this particular week things clicked and he was feeling more of the full force of it. He felt very afraid, and didn't know what to do.
He asked a mentor for advice. She asked "What does the fear feel like?"
He didn't articulate anything very clearly.
She followed up with "Does it feel more like a tiger is chasing you, or more like you want to curl up and hope that your parents come and fix it?"
"The second one."
She nodded. "I think that kind of fear does make sense in the ancestral environment (and often today). Sometimes you're afraid and it is the right thing to do to get help, and maybe curl up to conserve resources meanwhile.
"The thing is, though, that your parents really can't help you here. Mom can’t stop x-risk. So, that kind of fear... it just isn't very useful."
I can imagine people for whom this would be exactly the wrong thing to say. But, my friend nodded, and... something subsided in him. Fully understanding what the fear was for, and why it wasn't helpful, allowed him to process and let go of it.
"Fully understanding an emotion such that you can let it go" is actually a separate topic, which deserves it's own post. For now, I'm bringing this up because the move of: "Oh, my [parents/friends/society] actually really won't save me here, I have to somehow change something inside myself instead of waiting, or trying to change something outside myself" is an important move to be capable of making sometimes.
This played a role in the first Major Grieving that kicked off my interest in Deliberate Grieving – I thought someone had wronged me. I wanted my peer group to somehow decide either that yes, they had wronged me, and express that judgment. Or, judge that "actually, so-and-so really didn't actually wrong Raemon, this is Raemon's job to adjust to." Or something.
But, this was during the pandemic, and my peer group was busy dealing with their own problems, and it really didn't make sense to call a tribunal about it.
What if social reality just wasn't going to issue a verdict here?
What would I do?
In that case, the thing to do was grieve. Recognize that yes, there was some legitimate reason that having Justice here would be right and good, but that it wasn't going to happen. And let go somehow.
More recently, I was dealing with an exhausting series of problems, and talking to a friend who was experiencing a very similar set of problems. I found myself saying "I want you... I want you to say that I'm Good. To say that I've done enough."[6]
And my friend sadly said "Yeah. I also want someone to say that to me, Ray.
"The thing is, getting that from outside isn't ever really enough. I think you need to somehow become the sort of person who decides for themselves whether they're good enough, and to not need it from some kind of external validation."
Fuck.
What if there would never be someone I trusted who could tell me I was Good Enough, that things were in some sense Okay?
What if I had to learn to do that myself?
Or, worse, what if I had to really acknowledge that it was the wrong question to ask?
I think, for many people, it's important at some point to gain a true sort of independence, the way these fables and anecdotes gesture at. The Shadowmoth Struggle is still, often, a necessary part of reality.
The transhumanist rejoinder is still important. Sometimes your friends can help you. Sometimes you can invent better pedagogical techniques, or just actually tell someone they're Good Enough in a way that lands and that their soul really believes and updates on in an enduring way. We're social creatures. I'm pretty sure my human values ultimately say: it is good for people to help each other in this way.
But, I think it is also a worthwhile goal, to be the sort of person capable of enduring and growing and striving, even if at the moment you are an island.
Dig through the cocoon, such that your wings might make music in the skies of distant planets.[7]
It matters not how strait the gate,
How charged with punishments the scroll,
. I am the master of my fate,
I am the locus of control.
– William Ernest Henley and Kenzie Amodei
The metaphor of "moths struggling to escape a cocoon and needing to do so on their own" also shows up in the TV show Lost, but, in the relevant episode, the metaphor ends here, right before we get to the important bit.
The second phase of his training is also fairly interesting. The rest of the book was mostly forgettable (in that I literally have completely forgotten it), although the very last sentence was also important to me so maybe it's worth it for that.
It's left deliberately unclear – one of the things I liked about the story is that it's one of few stories I've read which do not spell out the moral, it just presents a bunch of things that happened, and leaves the reader to draw their own conclusions about it.
(See also this comment by Tsvi about why to be wary of this)
I perhaps want to temper all this with "I'm not sure how well this works – I believe it has helped me, but is not at a point where it's very legibly justified, and it required quite a lot of hours which others haven't replicated"
I want to add: I have had people tell me this, and it was helpful. Or otherwise help my struggles feel "seen" and respected. But there was something impermanent and incomplete about it.
(But please do not kidnap and torture people even if you're pretty sure it'll help them self-actualize)