You learn the rules as soon as you’re old enough to speak. Don’t talk to jabberjays. You recite them as soon as you wake up every morning. Keep your eyes away from screensnakes. Your mother chooses a dozen to quiz you on each day before you’re allowed lunch. Glitchers aren’t human any more; if you see one, run. Before you sleep, you run through the whole list again, finishing every time with the single most important prohibition. Above all, never look at the night sky.

You’re a precocious child. You excel at your lessons, and memorize the rules faster than any of the other children in your village. Chief is impressed enough that, when you’re eight, he decides to let you see a glitcher that he’s captured. Your mother leads you to just outside the village wall, where they’ve staked the glitcher as a lure for wild animals. Since glitchers are too slow and uncoordinated to chase down prey, their peculiar magnetism is the only reason they’re able to survive in the wastes.

Each of the glitcher’s limbs is tied to the ground. Its clothes are rags by now. As the group gathers around it, it starts to moan through its gag, a painful undulating noise. “Look at it,” Chief says. As if roused to a frenzy by his voice, the glitcher throws its body from side to side, shaking against its restraints. “This is what you’ll end up as, if you’re careless.”

Suddenly one of the glitcher’s arms breaks free. It waves in the air, fingers forming frantic spasmodic patterns. You stare at it for a second, before your mother yanks you around and buries your face in her side. When she lets you look again, two men have wrestled its arm back into place. Chief looks at you somberly. “If you’d kept watching for a few more seconds, you would have been hypnotized. And if you’d stayed hypnotized for a minute, even odds that you would have glitched yourself. That’s how easy it is to be careless. Do better, or you won’t make it to adulthood.”

You have nightmares for the next few days, your mind full of the glitcher’s slack face and its writhing fingers. You lull yourself back to sleep by reciting the rules. You’re determined that you won’t mess up again. And so you make it all the way to thirteen before everything goes wrong.

It’s morning on an ordinary day. You left your room to wash, and when you walk back in there’s a screensnake curled up in the corner. You look away immediately, but there’s a second one crawling towards you from the side, and your eyes lock onto it. You freeze for a moment, not knowing where else to look—and that’s long enough for the patterns on its skin to catch your gaze. They flicker, blooming in radiant colors. There’s something hypnotic about them, and for a few seconds you can’t look away.

Then an axe comes down, and you hear voices shouting, and a piercing scream. You blink, and shake your head muzzily, and when you look up again Chief is throwing a blanket over the screensnake corpse.

“Fuck,” Chief says. “God fucking damn it.”

“What happened?” you ask. There’s a gasp from the doorway; you turn to see your mother. “He’s okay! He’s okay he’s okay he’s okay—” She starts towards you, but Chief moves faster, stepping in front of her and pushing her back.

“Think! It got him just as he walked in. How long ago was that—five minutes? Ten? That’s a lethal exposure.” His head never turns away from you as he says it, though his eyes are focused over your shoulder. He’s still holding his axe in his hand.

“But he can still talk! No glitcher can talk!”

“Nobody’s ever survived five straight minutes of screensnake trance either.”

“I feel fine,” you break in. “It wasn’t that long, was it? Or maybe I’m immune.”

Your mother lets out a sob. “See? He’s still thinking straight! He must be immune somehow, he must. Here, show Chief—”

“Quiet!” Chief barks. He backs out of the room, pushing your mother behind him. “Don’t go anywhere, child. And don’t say a word. I need to think.”

They leave you alone for hours. Outside, you can hear your mother arguing with the guards Chief has posted at your door. But they barely reply, and she gets nowhere. Eventually, you hear Chief’s voice outside again through the door. “Say yes if you can hear me; don’t say anything else.”

“Yes.”

“I’ve heard rumors that occasionally people arise who have some kind of immunity. But that’s all they are—rumors. And I will not risk my village on those. You can’t stay here.”

You hear your mother’s voice raised in a moan outside, but Chief’s voice cuts through it.

“One thing the rumors say is that there’s a whole village led by someone who can resist glitching. They say it’s two week’s travel north. I’ll give you enough supplies to get there. Maybe they’ll take you in, maybe they won’t. But either way, you leave today.”

His voice softens. “You were a good kid. I hope the rumors are right, and I hope you make it to safety. But whatever happens, you can’t come back.”


The deep wastes are quiet, and lonely. You’ve only ventured into them once or twice before. Now they’re all that you can see in every direction as you walk, following an old wives’ tale that’s your only remaining hope. Somehow, you’re less scared than you would have expected. You never really thought you’d be glitched, but it’s still a fact of life: you lose friends every year. And you are immune, you must be: you’ve felt totally normal since the screensnake attack.

You still don’t want to take risks, though. Each night, as the stars come out, you cover your eyes firmly, and keep them covered until dawn. On the third day you see a glitcher shambling towards you from afar, but you give it a wide berth, and quickly leave it behind. You set traps every night, but all you catch are two ratlings. Still, they help stretch out your dwindling supplies a little longer, giving you a little more room for error.

After ten days of walking, you start looking for signs of the village Chief had spoken of. There are often still roads leading to them—sometimes covered in sand, but visible if you’re careful. You scout in a zigzag pattern, trying to cover as much ground as possible. But you see nothing. In the back of your mind you start to wonder if Chief had just made up the story wholesale, to get you to leave quietly. You curse yourself for a fool, but keep searching.

Two days later, you stumble across a suspiciously straight line of sand dunes. You start digging at their base, and after a few minutes you spot the telltale dark gray pattern of a buried road. You follow it north-east, searching for any sign of human presence. The next day you start spotting traps. They’re mostly empty, but even when they’ve caught a thylac or jabberjay, you leave them undisturbed. A few hours later you see who’s been laying them: at first faint figures in the distance, slowly resolving to two men as you walk closer. They’re focused on their task; only when you’re a hundred meters away does one look up and see you.

They shout in alarm, and scramble for their weapons. You rush to reassure them, but they’re wary—you need to yell back and forth for a few minutes to convince them that you’re not a glitcher or a mirage. Eventually they agree to take you back to their village, though they first bind your arms behind your back.

After an hour of walking, you reach their village walls. The hunters confer with the guards behind the gate. Finally two guards grab you and pull you through the main street, to an imposing building larger than any in your village. They lead you to a room near the entrance, where an old man sits at a desk, writing.

“Shaman,” one of the guards says, bowing his head. “We found him wandering in the wastes. He said that he seeks refuge, and that he has information he needs to tell you personally.”

Shaman turns his head towards you. He stares at you for a long moment, then gestures for you to speak.

“Greetings, Shaman,” you say deferentially. “I traveled here because I heard that you are immune to being glitched. I discovered recently that I am too. If you allow me to stay, I will contribute to your village in any way that I can.”

Shaman’s eyes are cold. “Are you sure that this story is the one you want to stick with?” You nod. He turns to the guards. “Test him,” he says.

The guards pull you towards through a corridor, into a large, dimly-lit room. In it are more glitchers than you’ve ever seen in your life—each tied down to a table, twitching intermittently. The guards tie you to one of the empty tables in the same way, and gag you. They each grab a pair of bulky earmuffs, and carefully place them over their ears. Then they walk around to each of the glitchers and remove their gags, one by one. As they do, the room gradually fills with their moans.

All night you listen to the babbling of the glitchers, noises that sound too alien to be produced by human mouths. As you sleep, you dream that you’re a glitcher too, prowling across the wastes under a sky that you’re still too afraid to look at. In the morning the guards come back, with Shaman behind them. He motions, and they undo your gag. “Well?” he says.

“I’m unharmed. But you can test me further if you like,” you say.

Shaman’s eyes widen. “You were telling the truth, then.” He pauses, and smiles. “I’d almost given up hope. This makes you the very first to succeed.”

They treat you very differently after that. You’re given food, and a room, and several days to rest. You spend the time exploring the village, which is much larger than your own, and much more raucous. The people in the streets seem less scared than those you grew up with. You wonder if that’s Shaman’s influence, though you’re too wary of offending them to ask.

A few mornings later, Shaman summons you to his office again. At his gesture, you sit in front of him. He waits for a few minutes before speaking.

“Tell me, child. Have you ever wondered why the world is like this?”

You frown. “The stories say that things used to be better. The land used to be fertile everywhere, not just where we cultivate it. Animals used to be safe. Even the stars used to be a beautiful sight. And then… I guess there was some kind of terrible accident, though I’ve never heard any explanation of what it was.”

Shaman laughs. It’s an ugly sound. “An accident? An accident that breaks the sky in ways that go on to break humans? An accident that turns animals themselves into weapons against us? No, this was no accident. It was a deliberate, targeted attack.”

“But… for that to be deliberate… it requires unimaginable power. What sort of beings have that?”

Shaman nods. “That’s the right question. Or rather, half of the right question. The other half is: with so much power bent on our destruction, why do they not simply crush us like ants?”

“Oh.” You think for a moment. “They want to… drag it out? They want to torment us?”

“Possibly, but I don’t think that they care about us even enough to enjoy hurting us. No, my guess is that they work under constraints that are invisible and maybe even inconceivable to us. I think that there was some kind of bargain. Humans used to have power, real power. We negotiated with them in aeons past, setting up compacts that would protect us from direct attacks.

“But they’ve been getting around the rules, step by step. They figured out how to overwrite our minds with only a few minutes of visual stimuli, to reform us into vessels for their purposes. Not easily, and not precisely. But they don’t need to be precise. If they can glitch enough of us, time will take care of the rest.”

The sheer scale of what he’s saying overwhelms you. “So we’re doomed.” You feel a lurch in your stomach as you say it.

He shakes his head. “No. We can adapt too. We have adapted—with all our safeguards, all our rules. And we can learn from the techniques they use. I’ve been studying glitchers for a decade, but it’s been slow work. I need someone else who’s immune to help me run more experiments.”

He stands and leads you towards the room full of glitchers. As you enter, he grabs a book from the desk by the doorway.

“The first question is: can we make any sense of their language? Is it even a language at all? They occasionally commune with each other, and sometimes their actions are suspiciously coordinated, which makes me think the answer is yes.”

He flips open the book, showing you pages upon pages of scribbled notes and tallies. “When I listen to them, there are some repeating syllables, and some structure. Your first task is to replicate my observations, and see if you can make any sense of them yourself. It won’t be easy, but from there we might be able to find patterns that give us hints about how to make more people immune, or maybe even find a cure for those who have already been infected. Will you work on this for me?”

You feel dwarfed by the magnitude of his ambitions. It’s like you’ve been living in a cave for your whole life, and now you’ve suddenly emerged into the blinding light of the sun. You worry that your voice will break if you try to speak. But you nod, and he seems satisfied.


You spend weeks working with Shaman, then months. His mission consumes you. He saved you from a drawn-out death in the desert. But more than that—he’s given you a way to fight back against the sheer senselessness of the world, to strike a blow against whoever caused all of this to happen.

Most days you sit in the corner, taking notes, as Shaman runs through experiments. Many are attempts to uncover any lurking remnant of humanity within the glitchers. He makes them try to pull clothes over their twitching limbs, or vocalize human language again with their writhing tongues. If they don’t succeed, he hurts them. They’ve lost almost all of their minds, but they still understand pain.

When Shaman is busy, you go to the lab alone, and try to replicate his old experiments. When you’re tired of that, you sit and watch the glitchers’ indecipherable hand gestures, and practice mimicking them. Sometimes they seem to respond to you, but you’re not sure how much of it you’re imagining. The ones that Shaman has had for longer do seem cleverer, though. So you focus on them, slowly training them to follow your commands.

You meet others from Shaman’s village, but they seem wary of you. You don’t blame them. They’re terrified of glitchers, of course, and you spend so much of your time with them that you’ve even started to smell like them. But you don’t care for their company either. They have no idea how important your work is; nor can they discuss it intelligently even when you try to explain it.

So you spend most of your days with only Shaman and the glitchers for company. Sometimes you despair of ever making progress. But other days it feels like you’re communing with them, that you’re right on the cusp of understanding them. You go through those days in a fugue, only half-aware of what you’re doing. Your notes on those days are eerily insightful, though. And it often seems like Shaman is in the same fugue state—he works like a man possessed.

One night, you dream again of the glitchers, and when you wake up you find yourself in their room, listening to them. You flinch, and realize that you must have sleepwalked there. The glitchers’ heads are all turned towards you. God, that’s dangerous. You start to tie yourself to your bed at night, to prevent any accidents.

But you don’t want to take a break. With your help, Shaman is making more progress than he has in years. The two of you have noticed a kind of correspondence between their words and their gestures, and Shaman thinks it might hold the key to translating the glitcher language. You still don’t know what any of it means, but after a few weeks of practice you can listen to the glitchers’ mumbles and effortlessly trace out the corresponding patterns with your fingers. Shaman watches you intently. “Could you learn to speak like them too?” he asks you one day. “I think so,” you tell him. He nods with grim satisfaction, and you redouble your efforts.

Sometimes you think about your home village—whether your mother still grieves you, whether Chief regrets his decision, whether your friends are still following the same routines and playing the same games as they used to. Sometimes you wonder what your life would have been like if you’d stayed. But you don’t regret any of what happened—your work now is too important. So you sleep, you wake, and you sleep again, the days all blurring together—

You snap awake. You’re standing in the lab. Shaman is holding your shoulders and shaking you. His face fills your vision, twisted into a rictus of terror. “Listen. Listen! I thought I was immune too at first. But there’s no such thing. It’s a trap! You and I are just a new type of glitcher—subtle enough to blend in, smart enough to research how to create more of our kind. All our work, all our experiments, we’re doing exactly what they want. Destroy it all and kill me! Please, kill me now!”

You stumble backwards in shock, hitting the desk behind you with a thump. He flinches at the sound. For a moment a look of blank incomprehension appears on his face.

Then his eyes snap back into focus, and his voice mellows. “Forgive me. I’m an old man, and my mind sometimes wanders. What were we talking about?”

You stare at him. “What do you mean? You just told me that all our work is helping the enemy! You asked me to kill you!”

He smiles slightly. “It sounds like you’ve been having a bad dream. Go back to sleep. We can talk about this in the morning.”

The smile is what convinces you. Your hand goes to your dagger. As you lunge towards him, he moves backwards, almost in slow motion. You kill him quickly, mercifully. As soon as he stops moving, you stride over to his desk and write out a note. You keep it as brief as possible, to prevent any further contamination. “Immunity is a lie. He and I were just more subtle glitchers, and our research would have created more of them. Kill the captive glitchers NOW.”

You think about killing the glitchers yourself, but you’re worried that they’ll make enough of a fuss to rouse others. And you don’t trust yourself in their presence any more. You don’t know how close you are to being trapped inside your own body, like Shaman was. So you gather all the papers from Shaman’s desk in a bundle under your shirt, and walk quickly out of the building. Nobody sees you as you navigate to the edge of the village and scale the wall to the outside world.

For an hour you walk deeper and deeper into the desert, eyes fixed on your feet. Eventually the adrenaline wears off, and you find yourself shaking from the cold. This will have to do. You drop to your knees on the sand, and use your hands to dig a small hole in front of you. You pull the papers from under your shirt and pile them in. Your whole body is trembling still, so it takes you several tries to set them alight. Once you do, though, they burn merrily.

The dancing of the flames is peaceful, almost hypnotic. By the time it dies down, what happened in the village almost feels like a bad dream. You look around at the sand stretching out towards the horizon. It’s all so peaceful, so serene. Did he really say those things to you? Was it all some fevered imagining? You don’t know what to believe. But it doesn’t matter any more—you’ve burned your bridges. And if you can’t solve the problem of the glitchers, as you’d so fervently hoped, the only thing left is to make sure you don’t exacerbate it.

Your dagger is by your side as always. As you unsheath it, you notice that it’s still sticky with Shaman’s blood. Somehow that feels appropriate. You close your eyes, take a deep breath, then drive it into your chest. For a second you stay there, frozen—then, involuntarily, you slump onto your side like a broken doll. You feel your blood start to pool under your body. With the feeling of helplessness comes a feeling of release. Wonderingly, you realize that at last the rules no longer apply to you. You can do anything you want.

With a last spasmodic effort, you twist yourself onto your back. The night sky fills your sight for the first time, and you let out an involuntary sigh. It’s grander than anything you’ve ever seen. The stars, multicolored, whirl in patterns, dancing across the sky as you watch. Hypnotized, your focus zooms in and in, chasing the universe as it spins towards the center of your vision. Your last faint thought—oh. It’s so beautiful. Then your mind falls into the spiral, and you are lost.

New Comment
38 comments, sorted by Click to highlight new comments since:

One of the things I track are "ingredients for a good movie or TV show that would actually be narratively satisfying / memetically fit," that would convey good/realistic AI hard sci-fi to the masses.

One of the more promising strategies withint that I can think of is "show multiple timelines" or "flashbacks from a future where the AI wins but it goes slowly enough to be human-narrative-comprehensible" (with the flashbacks being about the people inventing the AI). 

This feels like one of the reasonable options for a "future" narrative. (A previous one I was interested in was the Green goo is plausible concept)

Also, I think many Richard Ngo stories would lend themselves well to being some kind of cool youtube video, leveraging AI generated content to make things feel higher budget and also sending an accompanying message of "the future is coming, like now." (King and the Golem was nice but felt more like a lecture than a video, or something). A problem with AI generated movies is that the tech's not there yet for it not being slightly uncanny, but I think Ngo stories have a vibe where the uncanniness will be kinda fine.

you'll lose an important audience segment the moment they recognize any AI generated anything. The people who wouldn't be put off by AI generated stuff probably won't be put off by the lack of it. you might be able to get away with it by using AI really unusually well such that it's just objectively hard to even get a hunch that AI was involved other than by the topic.

I'm skeptical that there are actually enough people so ideologically opposed to this, that it outweighs the upside of driving home that capabilities are advancing, through the medium itself. (similar to how even though tons of people hate FB, few people actually leave)

I'd be wanting to target a quality level similar to this:

Perhaps multiple versions, then. I maintain my claim that you're missing a significant segment of people who are avoiding AI manipulation moderately well but as a result not getting enough evidence about what the problem is.

I would bet they are <1% of the population. Do you disagree, or think they disproportionately matter?

both - I'd bet they're between 5 to 12% of the population, and that they're natural relays of the ideas you'd want to broadcast, if only they weren't relaying such mode-collapsed versions of the points. A claim presented without deductive justification: in trying to make media that is very high impact, making something opinionated in the ways you need to is good, and making that same something unopinionated in ways you don't need to is also good. Also, the video you linked has a lot of additional opinionated features that I think are targeting a much more specific group than even "people who aren't put off by AI" - it would never show up on my youtube.

Also, the video you linked has a lot of additional opinionated features that I think are targeting a much more specific group than even "people who aren't put off by AI" - it would never show up on my youtube.

For frame of reference, do regular movie trailers normally show up in your youtube? This video seemed relatively "mainstream"-vibing to me, although somewhat limited by the medium.

I don't fully agree with gears, but I think it's worth thinking about. If you're talking about “proportion of people who sincerely think that way”, and if we're in the context of outreach, I doubt that matters as much as “proportion of people who will see someone else point at you and make ‘eww another AI slop spewer’ noises, then decide out of self-preservation that they'd better not say anything positive about you or reveal that they've changed their mind about anything because of you”. Also, “creatives who feel threatened by role displacement or think generative AI is morally equivalent to super-plagiarism (whether or not this is due to inaccurate mental models of how it works)” seems like an interest group that might have disproportionate reach.

But I'm also not sure how far that pans out in importance-weighting. I expect my perception of the above to be pretty biased by bubble effects, but I also think we've (especially in the USA, but with a bunch of impact elsewhere due to American cultural-feed dominance) just gone through a period where an overarching memeplex that includes that kind of thing has had massive influence, and I expect that to have a long linger time even if the wave has somewhat crested by now.

On the whole I am pretty divided about whether actively skirting around the landmines there is a good idea or not, though my intuition suggests some kind of mixed strategy split between operators would be best.

phew, I have some feelings after reading that, which might indicate useful actions. I wonder if they're feelings in the distribution that the author intended.

 I suddenly am wondering if this is what LLMs are. But... maybe not? but I'm not sure. they might be metaphorically somewhat in this direction. clearly not all the way, though.

spoilers, trying to untangle the worldbuilding:

seems like perhaps the stars are actually projecting light like that towards this planet - properly designed satellites could be visible during the day with the help of carefully tuned orbital lasers, so I'm inferring the nearest confusion-generating light is at least 1au away, probably at least 0.5ly.

 it's unclear if we're on the originating planet of the minds that choose the projected light. seems like the buried roads imply we are. also that the name is "glitchers". 

 dude, how the hell do you come up with this stuff. 

 seems like maybe the virus got out, since the soft-glitchers got to talk to normal people. except that, the soft-glitchers' glitch bandwidth presumably must be at least slightly lower due to being constrained to higher fluency, so maybe it spreads slower..

 I do wonder how there are any sane humans left this far in, if the *night sky* is saturated with adversarial imagery. 

 I doubt this level of advers....arial example is possible, nope nevermind I just thought through the causal graphs involved, there's probably enough bandwidth through vision into reliably redundant behavior to do this. it'd be like hyperpowered advertising.

 but still, this makes me wonder at what point it gets like this irl. if maybe I should be zeroing the bandwidth between me and AIs until we have one we can certify is trying to do good things, rather than just keeping it low. which is also not really something I would like to have to do.

[-]gwern3015

I read the 'stars' as simply very dense low-orbiting satellites monitoring the ground 24/7 for baseline humans to beam low-latency optical propaganda at. The implied King's Pact presumably is something like, "the terrestrial Earth will be left unmodified and no AI are allowed to directly communicate or interact with or attempt to manipulate baseline humans", and so satellites, being one-way broadcasts outside the Earth, don't violate it. This then allows the bootstrap of all the other attacks: someone looks up at night long enough, they get captured, start executing the program. But because it's all one-way and 'blind', the attacks have to be blackbox, like evolutionary algorithms, and work poorly and inefficiently, and with little feedback. (If a glitcher doesn't work, but can only attract other animals rather than humans, where did your attack go wrong? How hard are you, bound by the King's Pact, even allowed to think about your attack?) The soft-glitchers are a bypass, a mesa-optimizer: you load the minimal possible mesa-optimizer (which as we know from demo scene or hacking can be relatively few bytes), an interest in glitchers, which exploits the native human intelligence to try to figure out an interpreter for the powerful but non-human-native (for lack of feedback or direct access to humans to test on) programs in the hard-glitchers. Once successful (ie. once they figure out what some ill-chosen gestures or noises were actually supposed to mean, fixing the remaining errors in the attack), they can then successfully interpret and run the full attack program. (Which might include communication back to the AI attackers and downloading refined attacks etc.)

Nice, that's almost exactly how I intended it. Except that I wasn't thinking of the "stars" as satellites looking for individual humans to send propaganda at (which IMO is pretty close to "communicating"), but rather a network of satellites forming a single "screen" across the sky that plays a video infecting any baseline humans who look at it.

In my headcanon the original negotiators specified that sunlight would still reach the earth unimpeded, but didn't specify that no AI satellites would be visible from the Earth. I don't have headcanon explanations for exactly how the adversanimals arose or how the earth became desolate though.

(Oh, also, I think of the attack as being inefficient less because of lack of data, since AIs can just spin up humans to experiment on, and more because of the inherent difficulty of overwriting someone's cognition via only a brief visual stimulus. Though now that I think about it more, presumably once someone has been captured the next thing you'd get them to do is spend a lot of time staring at a region of the sky that will reprogram them in more sophisticated ways. So maybe the normal glitchers in my story are unrealistically incompetent.)

My headcannon for the animals was that early on, they released viruses that genetically modified non-human animals in ways that don't violate the pact.

I didn't think the pact could have been as broad as  "the terrestrial Earth will be left unmodified," because the causal impact of their actions certainly changed things. I assumed it was something like "AIs and AI created technologies may not do anything that interferes with humans actions on Earth. or harms humans in any way" - but genetic engineering instructions sent from outside of the earth, assumedly pre-collapse, didn't qualify because they didn't affect human, they made animals affect humans, which was parsed as similar to impacts of the environment on humans, not an AI technology.

I appreciated this comment! Especially:

 dude, how the hell do you come up with this stuff. 

 

It took me several edits to get spoilers to work right, I had to switch from markdown to the rich text editor. Your second spoiler is empty, which is how mine were breaking.

I just thought through the causal graphs involved, there's probably enough bandwidth through vision into reliably redundant behavior to do this

Elaborate.

edit: putting the thing I was originally going to say back:

I meant that I think there's enough bandwidth available from vision into configuration of matter in the brain that a sufficiently powerful mind could find adversarial-example the human brain hard enough to implement the adversarial process in the brain, get it to persist persist in that brain, take control, and spread. We see weaker versions of this in advertising and memetics already, and it seems to be getting worse with social media - there are a few different strains, which generally aren't highly compatible with each other, but being robust to communicated manipulation while still receiving latest factual news has already become quite difficult. (I think it's still worth attempting.) More details:

According to a random estimate I found online to back up the intuition I was actually pulling from, the vision system transfers about 8Mbit/sec = 1Mbyte/sec of information, which provides an upper bound on how many bits of control could be exercised. That information is transferred in the form of neural spikes, which are a process that goes through chemistry, ie the shapes of molecules, which have a lot of complex behaviors that normally don't occur in the brain, so I can't obviously upper bound the complexity of effect there using what I know.

We know that the causal paths through the brain are at least hackable enough to support advertising being able to fairly reliably manipulate, which provides a lower bound on how much the brain can be manipulated. We know that changing mental state is always and only a process of changing chemical state, there's nothing else to be changed. That chemical state primarily involves chain reactions in synapses, axons, dendrites during the fast spike path, and involves more typical cell behaviors in the slow, longer-term path (things involving gene regulatory networks - which are the way most cells do their processing in the first place.)

The human brain is programmable enough to be able to mentally simulate complex behaviors like "what will a computer do?" by, at minimum, internal chain of thought; example: most programmers. It's also programmable enough that occasionally we see savants that can do math in a single ~forward-pass equivalent from vision (wave of spike trains - in the cortex, this is in fact pretty much a forward pass).

We know adversarial examples work on artificial neural networks, and given the ability of advertising to mess with people, there's reason to think this is true on humans too.

So, all those things combined - if there is a powerful enough intelligent system to find it (which may turn out to be a very tall order or not - compare eg youtube or tiktok, which already have a similar mind-saturating effect at very least when ones' guard is down), then it should be the case that somewhere in the space of possible sequences of images (eg, as presented in the sky), one can pulse light in the right pattern in order to knock neurons into synchronizing on working together to implement a new pattern of behavior intended by the mind that designed it. If that pattern of behavior is intended to spread, then it includes pushing neurons into processes which result in the human transmitting information to others. If it's far outside of the norm for human behavior, it might require a lot of bandwidth to transmit - a lot of images over an extended period (minutes?) from the sky, or a lot of motion in hands. In order for this to occur, the agency of the adversarially induced pattern would have to be more reliable than the person's native agency - which eg could be achieved by pushing their representations far outside of normal in ways that make them decohere their original personality and spend the brain's impressively high redundancy on

I'm guessing there aren't adversarial examples of this severity that sound normal - normal-sounding adversarial examples are probably only familiar amounts of manipulating, like highly optimized advertising. But that can be enough already to have pretty significant impacts.

what I originally said, before several people were like "not sharing dangerous ideas is bad", ish: I think I'd rather not publicly elaborate on how to do this, actually. It probably doesn't matter, probably any mind that can do this with my help can do it in not many more seconds without my help (eg, because my help isn't even particularly unique and these ideas are already out there), but I might as well not help. Unless you think that me explaining the brain's vulnerabilities can be used to significantly increase population 20th-ish percentile mental robustness to brain-crashing external agentic pressure. But in brief, rather than saying the full thing I was going to say, [original post continued here]

edit 12h after sending: alright, I guess it's fair to share my braindump, sounds like at worst I'll be explaining the dynamics I imagine in slightly more detail, I'll replace it here in a bit. sorry about being a bit paranoid about this sort of thing! I'm easily convinced on this one. However, I do notice my brain wants to emotionally update toward just not saying when I have something to not share - not sure if I'll endorse that, guessing no but quite uncertain.

in general I think people should explain stuff like this. "I might as well not help" is a very weak argument compared with the benefits of people understanding the world better.

It's a straightforward application of the Berryman Logical Imaging Technique, best known for its use by the other basilisk. 

Intuitively, I see a qualitative difference between adversarial inputs like the ones in the story and merely pathological ones, such as manipulative advertising or dopamine-scrolling-inducing content. The intuition comes from cybersecurity, where it's generally accepted that the control plane (roughly, the stream of inputs deciding what the system does and how it does it) should be isolated from the data plane (roughly, the stream of inputs defining what the system operates on.) In the examples of advertising and memetics, the input is still processed in the 'data plane', where the brain integrates sensory information on its own terms, in the pursuit of its own goals. "Screensnakes"/etc seem to have the ability to break the isolation and interact directly with the control plane (e.g a snake's coloration is no longer processed as 'a snake's coloration' at all.)

 

That said, there are natural examples which are less clear-cut, such as the documented phenomenon where infrasound around 19Hz produces a feeling of dread. It's not clear to me that this is 'control plane hacking' per se (for example, perhaps this is an evolved response to sounds that would have been associated with caves or big predators in the past) but it does blur the intuitive boundary between the control plane and data plane.

 

Are you aware of any phenomena that are very 'control plane-y' in this sense? If they existed, it would seem to me to be a positive confirmation that I'm wrong and your idea of the adversarial search resulting in a 'Glitcher protocol' would have some legs.

I think most things that hit your brain have some percentage of leaking out of the data plane, some on the lower end, some fairly high, and it seems like for current levels of manipulative optimization towards higher-data-plane-leaking media, looking for the leaks and deciding how to handle them seems to me like maybe it can help if you have to encounter the thing. it's just that, normally, the bitrate of control back towards the space of behavior that the organism prefers is high enough that the incoming manipulation can't strongly persist. but we do see this fail even just with human level manipulation - cults! I personally have policies like "if someone is saying cults good, point out healthy religion can be good but cult indoctrination techniques are actually bad, please do religion and please check that you're not making yourself subservient". because it keeps showing up around me that people do that shit in particular. even at pretty large scales, even at pretty small ones. and I think a lot of the problem is that, eh, if the control plane isn't watching properly, the data plane leaks. so I'd expect you just need high enough bitrate into the brain, and ability to map out enough of the brain's state space to do phenotype reprogramming by vision, michael levin sorts of things - get enough properly targeted changes into cells, and you can convince the gene networks to flip to different parts of their state space you'd normally never see. (I suspect that in the higher fluency regime, that's a thing that happens especially related to intense emotional activations, where they can push you into activating genetically pretrained patterns over a fairly long timescale, I particularly tend to think about this in terms of ways people try to get each other into more defection-prone interaction patterns.)

I'm not following how the cult example relates to something like achieving remote code execution in the human brain via the visual cortex. While cult manipulation techniques do elicit specific behavior via psychological manipulation, it seems like the brain of a cult member is still operating 'in human mode', which is why people influenced by a cult act like human beings with unusual priorities and incentives instead of like zombies.

I doubt the level of inhuman behavior we see in this story is remotely close to easy to achieve and probably not tractable given only hand motions as shown - given human output bandwidth, sounds seem needed, especially surprisingly loud ones. for the sky, I think it would start out beautiful, end up superstimulating, and then seep in via longer exposure. I think there's probably a combination of properties of hypnosis, cult brainwashing, inducing psychedelic states, etc, which could get a human's thinking to end up in crashed attractors, even if it's only one-way transmission. then from a crashed attractor it seems a lot more possible to get a foothold of coherence for the attacker.

Man, I really hope there's a way to induce psychedelic states through sensory inputs. That could be hugely beneficial if harnessed for pro-human goals (for example, scaling therapeutic interventions like MDMA or ketamine therapy.)

Given how spectacularly harmful psychedelic drugs can often be, I think we’d better hope that there isn’t any such “sensory-input-only” method of inducing psychedelic states.

edit: uh, well, short answer: there totally is! idk if they're the psychedelic states you wanted, but they should do for a lot of relevant purposes, seems pretty hard to match meds though. original longer version:

there's a huge space of psychedelic states, I think the subspace reachable with adding chemicals is a large volume that's hard to get to by walking state space with only external pushes - I doubt the kind of scraping a hole in the wall from a distance you can do with external input can achieve, eg, globally reversing the function of SERT (I think this paper I just found on google may show this - I forget where I first encountered the claim, not double checking it properly now), that MDMA apparently induces! you can probably induce various kinds of serotonin release, though.

but the premise of my argument here in the first place - where you can sometimes overwhelm human output behavior via well crafted input - is that that probably doesn't matter too much. human computational output bitrate seems to be on order ten bits per second across all modalities,[1] and input bitrate is way above that, so my guess is that update bitrate (forming memories, etc) is much higher than natural output bitrate[2], probably yeah you can do most of the weird targeted interventions you were previously getting via psychedelics instead from like, getting some emotional/tempo sorts of things to push into the attractor where neurons have similar functionality already. I just doubt you can go all the way to fixing neurological dysfunctions so severe that to even have a hope of doing it from external input, you'd need to be looking for these crazy brain hacking approaches we were talking about.

I guess what we'd need to measure is like, bitrate of self-correction internally within neurons, some FEP thing. not sure off the top of my head quite how to resolve that to something reasonable.

  1. ^

    of course, like, actually I'm pretty dang sure you can get way above 10bit/s by outputting more noisy output, but then you get bits that aren't coming from the whole network's integrated state. the 10bps claim feels right for choosing words or similar things like macroscopic choices, but something feels wrong with the claim to me.

  2. ^

    some concern I missed counterevidence to this memorization-bandwidth claim from the paper though!

Ok, thats mostly what I've heard before. I'm skeptical because:

  1. If something like classical adversarial examples existed for humans, it likely wouldn't have the same effects on different people, or even just viewed from different angles, or maybe even in a different mood.
  2. No known adversarial examples of the kind you describe for humans. We could tell if we had found them because we have metrics of "looking similar" which are not based on our intuitive sense of similarity, like pixelwise differences and convolutions. All examples of "easily confused" images I've seen were objectively similar to what theyre confused for.
  3. Somewhat similar to what Grayson Chao said, it seems that the influence of vision on behaviour goes through a layer of "it looks like X", which is much lower bandwidth than vision in total. Ads have qualitatively similar effects to what seeing their content actually happen in person would.
  4. If adversarial examples exist, that doesn't mean they exist for making you do anything of the manipulators choosing. Humans are, in principle, at least as programmable as a computer, but that also means there are vastly more courses of action than possible vision inputs. In practice, propably not a lot of high-cognitive-function-processing could be commandeered by adversarial inputs, and behaviours complex enough to glitch others couldn't be implemented.

for AIs, more robust adversarial examples - especially ones that work on AIs trained on different datasets - do seem to look more "reasonable" to humans. The really obvious adversarial example of this kind in human is like, cults, or so - I don't really have another, though I do have examples that are like, on the edge of the cult pattern. It's not completely magic, it doesn't work on everyone, and it does seem like a core component of why people fall to it is something like a relaxed "control plane" that doesn't really try hard to avoid being crashed by it; combined with, it's attacking through somewhat native behaviors. But I think OP's story is a good presentation of this anyway, because the level of immunity you can reliably have to a really well optimized thing is likely going to be enough to maintain some sanity, but not enough to be zero affected by it.

like, ultimately, light causes neural spikes. neural spikes can do all sorts of stuff. the robust paths through the brain are probably not qualitatively unfamiliar but can be hit pretty dang hard if you're good at it. and the behavior being described isn't "do anything of choosing" - it seems to just be "crash your brain and go on to crash as many others as possible", gene drive style. It doesn't seem obvious that the humans in the story are doomed as a species, even - but it's evolutionarily novel to encounter such a large jump in your adversary's ability to find the vulnerabilities that currently crash you.

Hmm, perhaps the attackers would have been more effective if they were able to make, ehm, reproductively fit glitchers...

Oh, something notable here - if you're not personally familiar with hypnosis, it might be harder to grok this. Hypnosis is totally a thing, my concise summary is it's "meditation towards obedience" - meditation where you intentionally put yourself in "fast path from hearing to action", ish. edit 3: never do hypnosis with someone you don't seriously trust, ie someone you've known for a long time who has significant incentive to not hurt you. The received wisdom is that it can be safe, but it's unclear if that's true, and I've updated towards not playing with it it from this conversation.[1] original text, which was insufficiently cautious: imo it's not too dangerous as long as you go into it with the intention to not fully yield control and have mental exception handlers, but doing that intention activation of your attention to not leave huge gaps in the control plane seems potentially insufficient if the adversary is able to mess with you hard enough. Like, I agree we're a lot more adversarially robust than current AIs such that the attacks against us have to be more targeted to specific human vulnerabilities, but basically I just don't buy it's perfect, and probably the way it fails for really robust attacks is gonna look more like manipulating the earliest layers of vision to get a foothold.

[1] Also, like, my current view is that things like the news or random youtubers might be able to do hypnosis-esque things if you approach them sufficiently uncritically. not to mention people with bad intentions who you know personally who are specifically trying to manipulate you - those keep showing up around these parts, so someone who wants to do hypnosis IRL who you met recently should not be trusted - that's a red flag.

for AIs, more robust adversarial examples - especially ones that work on AIs trained on different datasets - do seem to look more "reasonable" to humans.

Then I would expect they are also more objectively similar. In any case that finding is strong evidence against manipulative adversarial examples for humans - your argument is basically "there's just this huge mess of neurons, surely somewhere in there is a way", but if the same adversarial examples work on minds with very different architectures, then that's clearly not why they exist. Instead, they have to be explained by some higher-level cognitive factors shared by ~anyone who gets good at interpreting a wide range of visual data.

The really obvious adversarial example of this kind in human is like, cults, or so

Cults use much stronger means than is implied by adversarial examples. For one, they can react to and reinforce your behaviour - is a screen with text promising you things for doing what it wants, with escalating impact and building a track record an adversarial example? No. Its potentially worrying, but not really distinct from generic powerseeking problems. The cult also controls a much larger fraction of your total sensory input over an extended time. Cult members spreading the cult also use tactics that require very little precision - there isn't information transmitted to them on how to do this, beyond simple verbal instructions. Even if there are more precision-needing ways of manipulating individuals, its another thing entirely to manipulate them into repeating those high precision strategies that they couldn't themselves execute correctly on purpose.

if you're not personally familiar with hypnosis

I think I am a little bit. I don't think that means what you think it does. Listening-to-action still requires comprehension of the commands, which is much lower bandwidth than vision, and its a structure thats specifically there to be controllable by others, so it's not an indication that we are controllable to others in other bizzare ways. And you are deliberately not being so critical - you haven't, actually, been circumvented, and there isn't really a path to escalating power - just the fact youre willing to oey someone in a specific context. Hypnosis also ends on its own - the brain naturally tends back towards baseline, implanting a mechanism that keeps itself active indefinitely is high-precision.

your argument is basically "there's just this huge mess of neurons, surely somewhere in there is a way",

I suppose that is what I said interpreted as a deductive claim. I have more abductive/bayesian/hunch information than that, I've expressed some of it, but I've been realizing lately a lot of my intuitions are not via deductive reasoning, which can make them hard to verify or communicate. (and I'd guess that that's a common problem, seems like the sort of thing science exists to solve.) I'm likely not well equipped to present justifiedly-convincing-to-highly-skeptical-careful-evaluator claims about this, just detailed sketches of hunches and how I got them.

Your points about the limits of hypnosis seem reasonable. I agree that the foothold would only occur if the receiver is being "paid-in-dopamine"-or-something hard enough to want to become more obedient. We do seem to me to see that presented in the story - the kid being concerningly fascinated by the glitchers right off the bat as soon as they're presented. And for what it's worth, I think this is an exaggerated version of a thing we actually see on social media sometimes, though I'm kind of bored of this topic and would rather not expand on that deeply.

imo it's not too dangerous as long as you go into it with the intention to not fully yield control and have mental exception handlers

Ah, you're a soft-glitcher. /lh

Edit: This is a joke.

can you expand on what you mean by that? are there any actions you'd suggest, on my part or others, based on this claim? (also, which of the urban dictionary definitions of "lh" do you mean? they have opposite valences.)

edit: added a bunch of warnings to my original comment. sorry for missing them in the first place.

I meant "light-hearted" and sorry, it was just a joke.

Fair enough. Neither dill nor ziz would have been able to pull off their crazy stuff without some people letting themselves get hypnotized, so I think the added warnings are correct.

I love the title "Trojan Sky" and the word "screensnake".

Wildbow (the author of Worm) is currently writing a story with a quite similar premise

Awesome story! Don't have any big brain takes about what it "means", but I like the moment-to-moment descriptions of the world and the descriptions of the mental episodes the main guy is having towards the end. It flows well.

A nice scary story! How fortunate that it is fiction...

... or is it? If we get mind uploads, someone will certainly try to gradient-ascent various stimulus (due to simple hostility or Sixth Law of Human Stupidity), and I do believe the underlying fact that a carefully crafted image could hijack mental processes to some point.

Come to think of it, zebras are the closest thing we have to such adversarially colored animals. Imagine if they also were flashing at 17 Hz, optimal epilepsy inducing frequency according to this paper: https://onlinelibrary.wiley.com/doi/10.1111/j.1528-1167.2005.31405.x 

Curated and popular this week