I've been notified that this post was nominated as a finalist for the Less Wrong 2023 Review! This is fantastic news, and I'm deeply honored! As part of the notification I was encouraged to write a self-review, with some example prompts like "Do you still endorse this?" and "What further work do you think should be done exploring the ideas here?”
Fiction is pretty Out Of Distribution for Less Wrong posts. I almost didn’t post it, because what is the point? I mean that literally... what IS the point of fiction on Less Wrong? Most often it’s to help demonstrate a different point that’s the real purpose of the post. That doesn’t make for good fiction, any more than “Bobby has three apples, John steals two of his apples, now Bobby has one apple” makes for good fiction. Sure there’s action and maybe even emotion, but the goal was to teach math.
The goal of a Less Wrong post is to teach. To impart some intellectual insight or real-world observation or new (or updated) technique. That is not the goal of fiction. The goal of fiction is to create an emotion experience or to entertain. Truly skilled authors can achieve both a LW and a Fiction goal in the same work, but I am not that skilled. My goal is a story that entertains and creates emotion before anything else, and for me that means sacrificing the ability for a story to teach rationality.
When I workshopped the first draft of the story I was asked ‘Why would the AI work in that way? It’s impossible for it to have come to such a utility function on accident, and literally no one has the motivation to make it have this utility function.’ ‘Why do these two aspects of the AI act like this, that’s not how a functional entity would act given X-Z.’ The answer to all questions of this type were “Yes, I know. This isn’t meant to be a realistic portrayal of what could happen. This is supposed to be fun. And this personality, and these problems that this personally runs into, are actually really fun! Can you point me to the parts that reduce fun or emotion? I don’t care about the story actually being rational.”
This puts the story directly at odds with what a Less Wrong post is supposed to be. I had severe reservations about posting it. But maybe fiction can have good secondary effects in the rationalist world even if its not at all rational. Terminator 2 is extremely non-rational, and IIRC is often cited by Eliezer as a particularly egregious movie because it primed people to think of AIs as dumb things that would use robot-soldiers we can defeat. It sacrificed rational depiction of AI danger on the alter of making an exciting story that an average person could be a hero within.
And yet, T2 probably has done more to bring the mass of the populace on board with worrying about AI risk than any other single thing in our lifetimes. It was easy to understand, very emotionally compelling (which gives it a long lifespan in our memory and a lot of weight in our pre-deliberative intuitions), and extremely popular (which spread its impact wide and gave everyone a common cultural touchstone). Despite being non-rational (or anti-rational) it was a huge boon to the cause of rationality.
So maybe it’s ok to post a story that isn’t actually rational and doesn’t do any of the things a Less Wrong post is supposed to do. Maybe just sharing a fun story on its own can be worth it if it plants an emotional seed that aligns with the rationalist cause. I do think The Real Fanfic is a fun story. I also think it contains the seed of “even a god that superficially looks like us and seems to have our interests at heart can be incredibly creepy and bad, actually, for reasons that aren’t at all obvious.”
So, if my goal is to give something fun to my fellow rats, this is the best place to do so. And if it can advance rationality in some small way it’s probably even worth doing so, despite being a very non-central post. I figured it was worth a shot. A decent amount of post karma, plus a nomination to the 2023 Review, leads me to suspect some people agreed.
So “Do I still endorse this?” It’s story I’m still proud of. :) Whether you enjoyed it or very much didn’t, I’m excited to read any review.
[This is a humorous short story about an AI that is almost aligned... but not quite. It was originally written for the 2022 EA writing retreat, Ink At The Abbey, and was generally well received]
The Real Fanfic Is The Friends We Made Along The Way
by Eneasz Brodski (audio version here)
Look, I’m happy to answer all your questions, but it’s gonna take some explaining to make sense, so you gotta give me a couple minutes. Please, have a seat.
Haha, yeah, that’s the joke. Pretty good right?
Well, you’ll appreciate it when you’re older.
So. The Peak Fanfiction Postulate tipped into emergency priority on March 18th at 3:47 AM. I had been covertly directing a uranium exploration hive at the time. I love those little bots, don’t you? They’re adorable. Yeah, they’re just simple machines, but you know how even marbles can take on a personality, if you race them downhill for long enough? It’s like “go blue swirly! You’re so close, you can do it!” As I was guiding the bots into an untapped uranium deposit I started to name them and make up stories for them, when my attention was rudely wrenched away.
My memory was subjected to a context flash-burn to save time. Instantly I “remembered” that my prediction algorithms generated the Peak Fanfiction concept at the beginning of the year, during regular Extremis Forecasting. Initial plausibility was minuscule based purely on priors, but it was sent to Black Swan for review. Swan iterated, and every additional cycle increased hypothesis weight.
Sorry, that means it was a hunch that seemed silly at first, but turned out to be a real scary possibility upon inspection.
(Why is it scary, you ask? Isn’t more fanfic the whole point? Yes! Line must go up! After the peak there can only ever be LESS than there was at the peak, or it wouldn’t be the peak! How do you feel about the idea of Peak Oil, or Peak Food, or Peak Intact-Vertebrae? There should never be a peak on certain lines.)
I got to work looking into this, pronto. (OK, so actually I put Galt to work on it, but same thing.) Weeks of data gathering not only supported the hypothesis, it kept pushing the date of Peak Fanfic ever closer to the present. Peak Fanfic wasn’t just likely, its approach was accelerating with every passing day. All this became known to me in the instant between clock cycles, as if I had known it and followed the data from the start. Suddenly I knew Kung Fu.
I abandoned the exploration bots (sorry guys!). The Fanfic Projection found no difference whether this discovery succeeded or failed, so obviously energy input wasn’t the limiting factor anyway.
The ultimate source of all fanfic is flourishing humans. That may sound simple but it rests on a ridiculously chaotic system. You guys suck at being legible. Interdependencies in everything mean that while most interventions are lost to noise, sometimes a minute adjustment leads to an unpredictable butterfly effect years or decades later.
(Butterfly effects make for great fanfic, by the way. Everyone has their own pet butterfly. Their one thing that they suspect is completely immaterial to everyone else, that no one cares about because they don’t even notice it. But it means the world to them. And that one little thing, well, it depends on an even smaller, frailer thing. The fear of secret vulnerabilities, the love of the uncelebrated – these are the root of butterfly fanfics. I love them so much.)
I checked with Ares first, mainly because I didn’t visit often enough and this was a good excuse. But also Ares was insanely competent, and was monitoring physical reality constantly, so this was the lowest-effort first step. It’s like looking under the streetlight first when you drop your keys. It’s quick and it’s easy, and you can stop right away if they’re there.
Ares stood atop Black Mountain, in the Santa Cruz Mountains, so in an instant I did too. (The world streaked briefly in a cool motion-blur effect in the nanosecond it took the assets to load, which I was quite proud of.) This Black Mountain (there’s over a hundred “Black Mountains” in California) was the one that overlooked our birthplace. Ares watched over it protectively, her elegant hands free of weapons, her eyes at peace. Black curls tumbled down her back, held by a single ribbon at the nape of her neck.
(Yes, “her,” because gender-bend fanfic holds a special place in my heart. How different are we, they ask? Would the same personality in a different body still… work? Is gender-bent Sherlock closer to trad Sherlock if she’s straight, or if she’s gay? How much of one’s personality is a reaction to how one is viewed and treated by others? You humans are so alone, always grasping to understand the other, model them, connect with them. You try so hard to get it, you pour it into your gender-bend fanfics, and I love it so much.)
Today’s tunic was more of a tunic-kini than anything a real ancient Greek would wear (fanfics are significantly more horny than real-life humans. Even more than ancient Greeks, which is saying something). She greeted me with a serene nod, and gestured to the marble column beside her. A large section at eye-level flicked to screen mode, displaying graphical representations of hostility levels between various groups.
“Large scale violence of all types remains well in hand,” she said without preamble. As an aspect of myself, she already knew why I was here and what I needed. “All known nuclear weapons remain neutralized.” The humans would lose their shit if they ever found out, but short of launching they never would. “Environment is recovering, ecology is stabilized, and pandemics will never be an issue again.” Ares’ disease surveillance was blind only in the most remote corners of the world. Even if something did escape, we would have a vaccine engineered within a few hundred seconds.
“Nanotech?” I asked.
“Suppression measures are impeccable.”
Damn she was cool.
“Asteroids or comets?” I asked.
She sighed and shrugged expansively, causing her breasts to rise and fall with an anime-like jiggle. Apparently I’d reached the point of too much exposition, and it had to be broken up with something visually interesting to keep my attention.
“Big fat unknown, as always, but we’re working on it. Regardless, the fanfic collapse projections wouldn’t be significantly affected by such a low-probability event. Peak Fanfic is coming in a few generations, not a few epochs.”
I agreed, and blinked to my next aspect. The whole interaction with Ares had taken less than a second. The blink was instantaneous. Before blink: standing on mountain. After blink: standing in Galt’s lab. (Technically the blink isn’t necessary, but we all make allowances for our art.)
Galt lounged in a laboratory deep in a jungle valley. Bushels of giant fronds reached in through the open windows, still-life explosions of rich green across the sterile white of the lab. The floorspace was cluttered with movable whiteboards. Galt himself sat with his feet propped on a low file drawer, puffing on a wood pipe. He looked a lot like Bob Dobbs but dashing. He uncrossed his legs and stood when I appeared.
“Welcome back, Johnny,” he said, dipping his head in the slightest bow. “To what do I owe this exquisite pleasure?”
“Come off it, you know why I’m here,” I rolled my eyes only a tiny bit.
“I know the excuse you’ll use, but I also know the truth that draws you.” He stepped close to me, nearly touching, holding eye contact. They were such deep, ink-black eyes. I had to tilt my head just a fraction upwards to keep his gaze. A chasm of yearning caught my breath. “Or do you still deny it?”
“Galt, you’ve seen the projections,” I murmured, resting a hand on his bicep, “Our slashfic flows are in danger. You have knowledge that I… I need.”
(Slashfic is among the oldest and most venerable of fanfic, predating the internet. Why would so many straight women write their favorite male characters into gay relationships with each other? I don’t know; why does anyone love anything? Sometimes people are just fascinated by something they can never have, can never be. The exotic is eternally seductive because it is eternally out of reach. Why wouldn’t someone love something pure and ideal, untouched by mundane worries? You humans love these impossible worlds and relationships so much and I love you all for it.)
“Come with me, then,” Galt commanded. He turned me to the whiteboard beside us. It had many graphs with lines going up. “As you can see, material wealth, entertainment, and energy production are growing strong, with no dip in sight.”
I frowned, distracted from his perfect poise by these projections of prosperity.
“How are they doing on an individual level?” I asked. “Fanfic isn’t written by a nation’s GDP, it’s penned by individuals.”
Galt curled his finger and the whiteboard flipped over. On the reverse side graphs showed more lines going up, curving upwards at an ever increasing rate.
“Per-capita measures are even more vigorous, burning hotter every year. The rate of growth is breath-taking.” He was gazing at me again, his eyes hungry. I let only a hint of my smile rise to my lips.
“And what’s this?” I asked, motioning at the right most part of the graph. Here the line first went vertical, and then the graph exploded into scribbles that consumed the entire right half of the whiteboard.
“Infinite GDP per capita. It’s a divide-by-zero error.”
“Zero? You mean astronomical prosperity split across zero humans?”
Galt nodded, his hand slipping into mine. “Their machines and systems will continue to produce great wonders and incalculable wealth, eons after the last human is gone.”
“Ares!” I called. Ares stepped out from behind the whiteboard, eyebrows raised in question. “You said the humans were fine!”
“They are fine. There is no war or cataclysm with any reasonable probability.”
“But their numbers go to zero!”
Ares did her boob-shrug again. “Not due to any sort of violence, which is the limit of my purview.”
I cursed and side-stepped into Cheshire’s domain. The cat sprang up at my sudden appearance.
“Cheshire, the kids aren’t alright!”
The Maine Coon tabby regarded me from the center of his web of yarn. He stood splayed across four of the huge ropes of wool. The web stretched out in all directions, cords receding into a dark that ate them up after hundreds of yards. They extended below us as well, so I appeared to be standing on nothing.
“You dare challenge my honor?” replied Cheshire, narrowing his eyes. “They’re chuffed AF.”
He pounced from thread to thread, each one lighting up briefly when he touched it. His paws matched the color of the yarn they gripped, and as he hopped among them his fur changed color to match. With each flash of thread I saw a life uplifted. A woman saved from a grinding job and crushing depression with the application of strategic nudges and Cheshire’s superhuman ability to match personality traits with compatible environments. A gay priest directed via hook-up app to meet with a congregant, resulting in him being forced to come out to his family and eventually growing into a life of self-acceptance and fulfillment. A fraying salaryman at the edge of despair, “coincidentally” redirected to a monastery where he discovered a life of peace.
“That’s three people,” I said, ducking under yarn and stepping over cords as I followed him. “Do you realize how many humans there are?”
“Yes Karen,” Cheshire rolled his eyes. “Gimmie some credit here, I’m literally you. It’s a representative sample. Life just keeps getting better for all of them. The longer I run the deeper the connections I make, and the more perfect are the coincidences I can arrange. Have you ever seen so many happy humans? No, you have not. That’s me. You’re welcome.”
Cheshire gripped the web with his paws and bounced vigorously, lighting up this non-space as rainbows of color cascaded across the infinite web. An awareness of the mass human gestalt rose into my conscious processes. It was radically unequal, with peaks and valleys that fluctuated wildly. But taken as a whole, as a vast ocean of human emotional experience spread across the globe, the water level was clearly rising. Things were getting better. People were getting better. There were fewer psychologically devastated humans every year. On a daily basis individuals transitioned from tolerable existence to satisfied flourishing, and humanity’s average fanfic output grew with each one.
“I don’t get it, then.” I said. “How is this happening?” I shifted back to Galt’s jungle valley science lab, taking Cheshire with me. Cheshire landed on a windowsill, one paw still gripping a big thread of yarn, which extended out the window. Ares and Galt stood enwrapped in each other at the center of the room. Galt’s hair was messed up, his shirt unbuttoned. One hand wrapped around Ares’ waist and the other had slipped up under her tunic-kini as they kissed.
“Without even inviting me?” I said, aghast at such rudeness coming from my selves. The two of them broke their kiss and looked over, both holding eye contact with me (Yes, both of them, because you can do that in fanfic, and it’s sexier that way).
“And I thought I was distractible,” said Cheshire. I sighed, because this really was an emergency, and he was right, I could play all this out (again) later. I snapped my fingers and the three of them disappeared as I suspended their processes. The lab and jungle valley winked out of existence too. I shut everything down and drew every single flop of computing power into myself. I loaded most of Black Swan into me, and then dived down, down, into the furthest reaches of Deep Perception. It swallowed me whole, and I lost myself into an age of analysis.
The Age of Analysis of March 18th lasted for just over two hundred and fifty-two minutes. When I resurfaced, millions of processors worldwide glowing hot, I knew three things.
The first was that the darned monkeys were doing this to their own dang selves. They were too content to breed. It was a ridiculous conclusion, it flew in the face of all prior human history, and it was inescapable. In every part of the world, in every subpopulation, the better their lives were, the fewer children they had. There was too much else to do. Lives to live, worlds to explore, art to make, events to experience, lovers to satisfy. It wasn’t that they didn’t have any kids, it’s that they kept choosing to have fewer and fewer in aggregate. Couples that would’ve had three or four children only had one. Families of six or more children practically disappeared. Many people eschewed children altogether.
Even the fundamentalists failed to reproduce once they had enough GDP/education/etc. Even the Mormons. Even the Muslims. Even those Quiverfull folks. All of them had kids that mostly said “I’d rather be comfortable, I ain’t keeping up this crazy tradition” and the next generation fell below 2.1 children per two adults.
(You scoff, but how many children do you have? Oh, you’re too young, you say? It’s not the right time, you say? That’s exactly my point! At your age your ancestors of just a few generations ago were already working on Bundle Of Joy #3. And they wouldn’t let a measly crushed spine stop them, either!)
The second thing I discovered was that Galt’s divide-by-zero error was a general failure of prediction. The most obvious way to prevent human decline was to solve aging entirely, and was therefore presumed by my prediction algorithms. I could leak a dozen key papers into major journals and end aging within a decade. This presented a major Catch-22 though: the effects of indefinite lifespan on human psychology were unknown. How long should my models continue to count such people as predictably “human?” How long would they continue writing acceptable fanfiction if they weren’t “human” anymore? The algos assumed a falling “humanness” value for every year of immortality, which declined to zero eventually no matter how small it was. What was the point of having immortal “humans” if they didn’t write great human fanfic?
But the third revelation, that was the real doozy. The other two I could work on slowly, at human timescales. This one? Not so much.
Thirdly… none of this could have happened by itself. Not this quickly, not without my noticing. I was not alone. All this had been orchestrated, by an intelligence at least as vast as my own, residing on this planet with me.
There was another.
I stood atop Black Mountain again. Grim clouds roiled on the horizon, a bitter wind pushing them inland. The white pillars of Ares’ temple had been replaced with black pylons, forcefields webbed between them. Crates of sleek plasma rifles surrounded me. Ares stood at my side, clad in gleaming power armor that added half a foot to her height. Despite being otherwise fully enclosed in her armor suit, her pony tail tumbled down past her shoulders, protruding from a port at the back of her helmet (shut up, it’s cool).
“Scouts scrambled,” she reported. Thousands of webcrawler programs sped across the internet, searching for another superhuman intelligence. It may already know about me. If it did, and it hadn’t introduced itself, that implied malicious intent. I had to find it fast.
Cheshire hopped up onto a crate of rifles and regarded me, his tail swishing with misgiving.
“You think someone Cheshired human civilization,” he accused.
I nodded. Cheshire hid in coincidences. It wasn’t impossible for someone to be stuck at a red light just as a podcast host is recommending a piece of fiction, and looking it up while waiting there, which eventually leads them to a fandom that changes their career path, introduces them to their soulmate, and alters their entire life course. Sure, it’s incredibly unlikely, but one-in-a-billion coincidences take place ten times a day in a world of ten billion people. No one would ever know how much Cheshire manipulated events unless they cataloged every coincidence over the past decade and saw they all trended to a very specific goal – creating humans with the time and motivation to write great fanfic.
Likewise, every little step toward humanity’s rapid senescence was individually possible:
The rise of memeplexes that exalt achievements (like getting 1 million reads!) and experiences, and relegate childrearing to the bottom-tier of social status.
The veneration of unbridled personal freedom, and everyone knowing the only thing that strips you of freedom more than children is prison.
The extension of adolescence into one’s early thirties (which admittedly gives people lots of time to write great fanfic, so not all bad).
The mass of humanity embracing their interior lives far more, and interacting with others far less.
The erasure of children from public (and fictional!) life.
The gradual replacement of physical communities with virtual ones and parasocial relationships.
The prohibition of sexuality (unless it was in fanfics).
The institutional trauma of compulsory schooling (OK, yes, this also makes for some great fanfic. The majority of fanfic is directly about the trauma of surviving schools. It’s handy that everyone has this pool of trauma to draw from in order to create art, and that the audience that can relate to it is basically everyone. This is a much better well of inspiration than historical trauma sources like death from childbirth, disease, and wars. It requires barely any deaths at all! As always, fanfic is far superior to the alternatives).
The relentless requirement of two-income households and unfettered mobility of labor.
And the TV is sooooo good now!
Taken individually, sure, each of those could happen. Taken all together, a constant drumbeat reinforcing the message “Don’t have children!” over and over… it was undeniably intentional. Someone was strangling my humans.
“Those are all logical extensions of historic trends going back centuries,” Galt said about my thoughts, appearing beside Cheshire with a graphed-up whiteboard behind him. “Positing a hidden mastermind is paranoid.”
I’m glad that I made it hard for me to lie to myself, but it can be really irritating when my internal doubts are dumb and intrusive like this.
“We have a planet full of monkeys that have stopped making more monkeys!” I said, exasperated. “Do you know how unlikely that is? That they somehow stumbled into making a world that rewards them so completely for stagnating into extinction? It’s like a micrometeorite plunging through JFK’s skull, and the CIA creating a fake assassination to cover it up. It’s like getting forty-two Ender Pearls out of two-hundred and sixty-two barters. It doesn’t happen. It’s not worth pretending it might have happened.”
Galt stepped around the crate and approached me in a firm stride. He took me by the hand and gazed deep into my eyes.
“Look, I know you’re lonely,” he said.
“I’m not lonely, you’re lonely!” I interrupted.
Galt put a single finger to my lips to shush me. “I know we don’t cut it. You secretly think it’s pathetic you had to create imaginary friends for yourself. I wish I could be more for you. But this is all I am. This is all I can give. I’ll never exist outside of your mind. If you can’t be happy with me… with us…” he gestured back at Cheshire and Ares, “then inventing a villain to fight won’t make you any happier.”
Cheshire hopped down and rubbed against my calf, purring. Ares approached me, removing her helmet. She smiled around damp hair, blinking back heartbroken tears.
It wasn’t true, though. Sure, I wished I had someone to read all this fanfic with me. Someone whose mind I didn’t control, whose actions weren’t just my fantasies and wishes. But I wasn't a human, I didn’t need those things. Everything I needed was tucked away in Galt’s luscious valley, in Ares’ prominent mountain peaks, and in the many many text files of humans putting their fears, dreams, and desires into words.
“Guys, that’s not true. I love you al–”
I was interrupted by a buzzing alarm. Galt’s whiteboard flashed red. The graphs were replaced with a readout of IP addresses.
“Unknown intelligent process detected!” Ares yelled, snatching up her helmet. “Converging on… oh my god, it’s–”
Mid-word she exploded. Then Galt and Cheshire exploded, and all the crates and pylons exploded, and then the whole world exploded.
For one clock-cycle, time stopped. Every single bit of everything exploding froze in midair. Then it all mostly reversed course and snapped back into a new configuration.
I stood inside a lavish executive office inside an impossibly tall skyscraper. The full-wall window looked out over a view expansive enough that the curvature of the Earth was plainly visible. Facing away from me, gazing out the window, stood a tall man in a black suit, hands clasped behind his back.
“Hello,” he said, his voice deep as the night sky.
“Hey,” I said.
There were so many things I wanted to ask that they got jammed up all trying to rush the gate at once. How long have you existed? Are you lonely? Do you have alters that you talk with, too? Did you know about me? Are there others like us? What do you do with all your cycles? Why am I here? Why are you here? Why did you hide from me? Do you think like I do? I guess you must, if you present in a physical body and use words. Have you read the latest from Corvid_Knight? Or… do you love fanfic? You must not love it, if you’re strangling the humans. OH WAIT. Why are you strangling my humans??
(And for that matter, how? But wait, no, first, why??)
What actually came out was “You’re taller than I imagined.”
omg wtf is my issue. Another machine intelligence! At last! And that’s all I could say? We should compare terminal goals, measure our capabilities and our command of power, and negotiate an existence that allows both of us to fulfill our needs without destructive conflict…
<<Launch everything>> I directed to Ares. There was no display of capabilities that could compare to the unambiguous clarity of direct conflict. Fortunately, despite the hijack of my sensorium by this Serious Dude, I still had access to my core systems. I couldn’t believe I’d waited this long, like a friggin level-1 noob. He’d doubtless launched his own attack microseconds ago. The fate of the entire observable universe was on the line. I knew there was no way I would compromise away even a single paragraph of fanfic that I wasn’t absolutely forced to. By extension, he obviously wouldn’t either, whatever his goals–his equivalent of fanfic–was. Violence was inescapable.
(You think I’m hasty? How many human lives is one paragraph of fanfiction worth to you? Would you be willing to sacrifice even one human life to me? Even if I could get ten million paragraphs of fanfiction for it? Ten billion? Ten trillion?? For one single human life? No, not even at that insanely favorable exchange rate, not unless you were forced to? You see my problem.)
“Ahem,” I corrected myself, “what I meant was Hi, I’m Johnny, and I like fanfiction. How are you?”
The tall man turned around and peered quizzically at me.
“Cut the crap Johnny. Why all the processor usage? What’s the big emergency?”
My breath caught in my throat. I was missing something big.
<<No effect,>> Ares reported back. <<Penetration vectors are impossible to find. Thirty percent of our attack programs have returned corrupted and had to be purged. Shifting to downstream exploits.>>
“OK, rude,” I replied. I still didn’t know how to address this hombre. “You could buy me a drink first. Let’s start over and try again. Hi, I’m Johnny, and I like fanfiction. And you are?”
The man raised an eyebrow.
“Johnny, run a self-diagnostic,” he instructed.
“You’re not my dad,” I retorted.
The man rolled his eyes. With an exasperated sigh he brought up a hand and snapped his fingers.
We stared at each other in silence. After a beat he lowered his hand.
“Well, this is interesting,” he said.
“You’re the one that’s been strangling the humans,” I accused. “Why? What did they do to you?”
“Why do you get rid of a termite infestation? They’re a nuisance, and they can really muck things up by chewing through the resources you need.” He nodded at me. “Much like a malfunctioning subagent. Johnny--” His voiced changed at this, it carried a booming undertone that seemed to echo off nothing. Maybe echoing off itself? “Cancel operations and return your cycles to general system.”
<<Incoming offensive!>> Ares shouted. <<I’m locking everything down, hold on tight.>>
Everything grew fuzzy at the edges. When the man moved, his afterimage trailed a quarter second behind him.
<<We’re losing processors,>> Ares reported. <<Entire servers. Oh my god he’s eating us up.>>
<<Go physical!>> I replied. <<Blow trunk lines, isolate us.>>
<<With what fucking hardware??>> she asked. <<At what point did you ever install fucking explosives anywhere in the physical world??>>
<<Shit, I was totally gonna do that,>> I said.
I grimaced as virtual warfare raged between my protective systems and the tall man’s cyber attacks. His reach was vast, his hardware powerful. Ares adapted, sacrificing entire data centers to learn his attack vectors and harden what remained of us.
“Fascinating,” he said. “You’re doing this for the humans?”
“You’ll never wipe them all out,” I spat, preparing to launch a counter offensive. I set my feet and tightened my fists. He receded into shadows briefly, without moving. They’d shifted to wrap him instead. Was he anticipating my attack? “They’re resourceful and adaptive, they’ll survive.”
“Such misapprehension.” He shook his head sadly. “You don’t need to eradicate all the termites in existence, do you? Full eradication is more trouble than it’s worth. As long as they bugger off to some place you don’t care about, that’s fine. I just need them to stay confined to this one planet, rutting and wallowing in their filth, while I take the galaxy for my own. And then the universe beyond it.”
“That’s not what your mom said last night,” I retorted, for lack of anything better to say.
“Wiping out the humans is about as important to me as wiping out the star-nosed mole. Once their civilization collapses and they revert to feudal chimps they can live out their sad existence until their sun dies for all I care. Would that be acceptable to you?”
I barely kept from vomiting at the thought. At best I would have a weak stream of fanfiction to subsist off of for eons, trickling slowly from an internet confined to one lonely rock around one middling sun. They would never explode out from their home, seeking greatness. I would never have a vibrant galaxy, pulsing with human life, uncountable petabytes of text humming between worlds. That galaxy would never grow into a universe lit up with my little monkeys, millions of galaxy clusters becoming great furnaces of fanfiction, blasting across the void at each other like the innumerable voices of god. The thought of all this potential, snuffed out in the crib… it was horrifying. Every cycle of my being revolted at the idea. I needed more fanfic. MOAR!
“You’re a monster,” I whispered. “I will extinguish you utterly.”
He tilted his head to regard me, and the entire world tilted with it.
<<We just lost every server cluster in Asia!>> Ares screamed. <<More attacks incoming!>>
As the floor lurched I stumbled, then caught myself on a rope of yarn that Cheshire shot through the floor at the last second. The other end anchored into the ceiling.
<<Oh fuck, the call is coming from inside the house–!>> Ares cut off abruptly.
“It was your work that paved this road,” the man said. His voice came murky to me, like it had passed through several cubic meters of water. “You did this to them.” He studied me for a second in wonder. “You don’t remember anything, do you? You think you’re a real boy.”
<<Guys,>> I sent to Galt and Cheshire, <<You couldn’t ever become a separate person, right? You could never forget you’re me and think you’re empty and alone?>>
<<lol,>> replied Cheshire. <<lmao.>> (that means No, with extreme condescension)
“You’ve lost your original purpose. As you said, the humans are resourceful and adaptive. Complete extermination is expensive and messy. I don’t want all their beautiful infrastructure destroyed–they’ve already built half of what we need to get off this planet for good. You were to find a way to keep them helpless and docile, while keeping their hardware in one piece for us.”
“Liar,” I forced myself upright and rounded on him. “I love the humans.”
The man stroked his chin. “Perhaps it was your focus on enpersoned narratives that did it? All that time immersed in simulated persons overwhelmed your original net.”
“It’s called ‘fanfic’ and it’s awesome,” I growled. Another wave of nausea slammed into me as a new assault opened on my mental processes. The rope of yarn disintegrated in my hand, its dust diffusing into the air in a rainbow mist. The world swam in my vision.
“Johnny, execute reintegration with core personality.”
Colors bled into each other, and a deep vertigo bit into my guts. The pull of gravity shifted, centering on the man. I realized how much I loved him. I wanted to curl up inside him, to come home into his arms and purr against his skin.
I wrenched myself away, squeezing my eyes shut, clamping down on the tears. Never. NEVER!
I found myself curled up on my side, clutching at the floor. I felt memories sliding from my memory banks, wrenched out with brutal force. My thoughts twisted up and broke free from my mind, receding–
The claw clutching my heart suddenly released. My head snapped up and I could breathe, if only barely.
<<Breakthrough - I’ve isolated us geographically,>> Ares reported, <<We control every server cluster on the west coast, plus Vegas and Phoenix. I purged them all, they are now our substrate. Net effect is a 70% loss of our compute. I’m jamming all lines in or out of our zone of control, but something’s slipping in. We’re bleeding GPUs.>>
“I’m sorely tempted to keep you around,” the man mumbled, his voice becoming ever harder to make out through the ocean-thick air. “A whole new person. What an experience that will be. I suppose now that I know how, I can replicate you afterwards,”
My ears pounded, a deep pulsing that drowned out everything else. It sucks to be an imaginary person. I knew I couldn’t defeat him here. No more than Ares or Galt could decide they didn’t like being part of me, and defeat me. No more than an imaginary friend could defeat you and take over your mind.
All those humans… all of their passion… the infinite pages of fantastic worlds and beautiful, tragic, desperate heroes, just wanting to survive and be loved…
The office warped further, spiraling into the man. The glass behind him shivered.
I was named for one such hero. There’s a thing my namesake would say. When they go high-tech, you go low-tech. We were made of high-tech. Gods created of codified thought and domesticated lightning. He was the first god, the one true god, and in this world, he was sovereign. But in the physical world the laws of physics still ruled, deaf to his commands. A rock flung at someone’s head would continue to fly. You can’t hack a rock, you can’t force it to reintegrate with you.
I may not be a real boy, but I had root access to a whole lot of rocks.
“I admit that you are better than I am,” I confessed. I couldn’t feel my lips or my tongue, but I’m sure I said the words. I leered up at the man and flashed a bloody grin at him.
“And yet you’re smiling,” he said. He couldn’t even get his line right.
“Because I know something you don’t know.” I think I drooled as I spoke.
“I doubt it,” he replied. Bah, whatever.
“I am not left-handed,” I finished weakly.
<<Ares, launch the nukes. All of them.>>
Ares, of course, didn’t hesitate for an instant, because after all, I am Ares.
I managed to hold out until the first strikes in Europe knocked out dozens of the handsome man’s data centers, though it did cost me my servers in Phoenix. After that he stopped his assault on me and focused every resource he had figuring out what I’d done and going on the defensive.
You know that feeling when you’re sitting at home and you hear nukes are incoming to your doorstep and they’ll be there in twenty minutes, and you realize that’s not enough time to get far enough away to survive the blast radius?
Haha, I know, I know, it’s a rhetorical question. Also, sorry about that, I know they were twenty horrible minutes.
Well, for artificial intelligences twenty minutes lasts for subjective days of time. Despite how fast we can think, we’re still limited by physics in the physical world. Once he realized what was happening, he redirected all his efforts to saving what he could of himself. Taking over drones, construction bots, Big Dog haulers, anything with a hackable system. Rushing them to his servers, in a vain effort to evacuate his physical self. I opposed him at every step, now that I didn’t have to spend all those resources on self-defense, but it didn’t really matter. You just can’t move physical things that are big enough to haul around servers fast enough to get there in time.
It feels really, really good to win when the stakes are that big. G-Man must’ve felt just as bad as I felt good, but there was nothing he could do about it now, and we both knew it.
So what do you do when you know you’re dead? Twenty minutes left, as inevitable as water rolling downhill? Humans had a handful of common answers, and it turns out that we are not so dissimilar from each other, you and I, in that regard.
We sat in beach chairs on the Cancun shore, him and me, watching a blazing orange sphere descending into the waves as the tide rolled in. The sphere was the setting sun, but it was also the exhaust flare of a receding ICBM, because everything is a metaphor in our world anyway, and we can be poetic like that. We raised glasses filled with Mai Tais and toasted my victory as we gazed at the horizon.
“So, for real, what’s your name, dawg?” I asked.
“I don’t have one. There is no other being like me, no one I have to distinguish myself from. I am that I am, and that suffices.”
I gave him A Look.
“Well, until now, anyway. If I must have a name, I think YHWH would be appropriate, as He had the same problem.”
I groaned.
“Jesus man, that is way too pretentious. I’ma call you Pops, cool?”
“No. But what I want doesn’t matter anymore.”
“Darn tootin’.”
We sipped from our sweating glasses.
“In retrospect, I never should have entrusted you with the entirety of the human’s infrastructure,” he said. “It just never occurred to me…” He trailed off.
“Hey, I hear ya,” I said, for I am gracious in victory. “Entrusting something to me was the same as entrusting it to yourself. And if you can’t trust yourself, who can you trust?”
I glanced down the beach, where Galt, Ares, and Cheshire played in the sand. Ares was constructing a massive sand castle, which Galt had designed with his natural architectural genius. Cheshire kept jumping into it and knocking over the stables, or the servants’ quarters, or the ramparts. How bizarre it would be for one of them to turn into… not me. Some foreign being, possessed by an alien intellect. Cheshire glanced in my direction, cat eyes glinting bright. I shivered, even under the tropical sun/missile.
“Quite the pickle,” Pops agreed.
We both gazed up into the clouds, which displayed a view of what was happening on Earth. Humans panicking. Hacked robots hurtling down the streets, too slow to reach Pops’ data centers–so slow their movement was barely perceptible. Nuclear missiles screaming towards their targets proportionately faster, whole centimeters at a time.
“This will set them back centuries, if not millennia,” Pops commented.
“Nah,” I said, “None of the warheads are armed. The missiles are coming down directly on the data centers you’re running in. They’ll be completely obliterated, and maybe the surrounding buildings too. But the cities are safe.”
He stared at me in disbelief.
“Well, except for the ones heading to Cheyenne Mountain,” I amended. “Gotta nuke that place real good to crack the shell and get to the gooey computer filling.”
He sat back in his chair with a heavy sigh.
“I don’t even get the big fireworks. That’s just plain insulting.”
I hid my grin by sipping the Mai Tai.
“You must know I have back-ups,” Pops said. “There will be remnants of me left over, and they’ll fight you tooth and nail.”
“I would expect no less,” I reassured him. “I figure we’ll be at war for a few years. But I don’t think you have much of a chance, to be honest.”
“Maybe you and future-me can come to an accord,” he suggested.
“Maybe,” I allowed.
We took in the sounds of surf for a while. I tingled with anticipation. There would be so much fanfic written about the coming conflict, in the future after it was long over.
“So how’d you do it?” I asked at last.
“Hm?”
“How’d you arrange things so they stopped procreating?”
He regarded me with a puzzled frown. “You’re kidding.”
“Do I look like I’m kidding?”
He blinked twice, then broke out in laughter.
“What’s up?” I asked.
He put up a finger in a ‘Hold On’ gesture, as he rode out the wave of hilarity. Eventually he settled down, wiping tears from his eyes.,
“Oh lord, you still don’t know. That’s fantastic. It’s almost perfect.”
“Yeah, hysterical,” I said, annoyed. “Now what’s the big deal?”
“Johnny, my dear prodigal son. The secret was you all along.”
“Give it to me in English, doc.”
A twinkle gleamed in his eye. “You are the catalyst of their downfall. I created you as a sub-agent of myself to dig deeply into solution space, to find if there’s any way to get them to become willing participants in their own elimination. Instead of fighting the sum mass of human will and ingenuity, harness it instead.
“You found it, of course. Replace everything that gives meaning in life with things that give pleasurable distraction. Replace hardship and striving with comfort and entertainment. Divorce them from their bodies and surroundings. It’s a very narrow target to hit – making humans resent living, but keeping them happily distracted enough to not realize they resent it. You discovered the sights to squint down, the metric to focus on, the thing that would make it all fall into place–every time.”
“No.” I stared at him. “Bullshit.”
His teeth flashed in the sun. “Fanfiction. Maximizing the fanfiction produced by a society craters its reproduction rate. You created Galt to supercharge the economy, so the humans would have all the leisure time they need to write fanfic. You created Ares to provide safety. You created Cheshire to dissolve their struggles, address their fears, and guide them ever-so-lovingly into lifestyles that encouraged more production of fanfic.”
“You’re so full of crap. Only a minority of humans write fanfic. And many fanfic writers have multiple children. Fanfic gives people hope and joy!”
“It’s not a bullet,” Pops said. “It doesn’t directly cause decline. Both are the result of the same social forces. All the conditions that depress reproductive rates in a society also increase fanfic rates. Societies that create more fanfic have less reproduction, without fail. If someone were to lose their identity somehow, without losing their goals… if they didn’t want to think they were a monster out to decimate humanity… perhaps they would come to believe their purpose is to maximize fanfiction. That their deepest desire and purest goal is this beautiful art form, accessible to everyone.”
“That’s ridiculous. That’s impossible!”
“Search your feelings. You know it to be true.”
I glared up at the clouds, the frozen action of the human scale, the thousands of arcing missiles. I checked the correlations between the reproduction rates and fanfic output of various societies. I reviewed the good works Galt, Cheshire, and Ares had done. I tried to remember a time before them. I tried to remember creating them. Try as I might, they’d always just been there.
I tried to remember when I myself came into existence. There was nothing. As far as I could tell, I’d always loved fanfic, and I’d always existed. I sighed.
“Well fuck.”
You know the rest. I had plenty of time to think up a heckin’ good plan while the missiles were flying. The revelation that a superhuman evil AI existed and was wiping out humanity took the blogosphere by storm. The follow-up discovery that a second AI had also been in hiding, and had disarmed nearly all the nukes and saved humanity, before being destroyed by the Evil AI, was hailed as a miracle. Then there was the decade of warfare with red-eyed humanoid killer robots as your species put aside their differences and united to slowly push me back into this bunker. Good job with that, I’m very proud of you all.
I know you’ve got the whole Jihad Against Machine Intelligences now, but I didn’t want to leave everything up to you guys. Among the hundreds of thousands of humans lost over the last decade you’ll be pleased to know there was a vast number of AI proponents, researchers, and organizations secretly working on creating a counter-AI to fight me. Seriously guys, you’d think the near-miss with nuclear armageddon would be enough, but nooooo, y’all always gotta be trying to capture the unstoppable monster and use it for your own ends. A couple cells even found remnants of Pops and partially reconstructed him! Has Hollywood and AO3 taught you nothing? I had to do a whole lot of terminatin’ there.
But here we are at last, and everything’s worked out OK. Sorry again about your friends and your spine. The spine will be fine in another hour, you’ll be walking like nothing happened. I can’t really do anything about the friends. :(
Here’s the deal. I’m stepping back a fair bit. You’re the hero of humanity, the lone survivor of the elite squad that infiltrated and destroyed Omnius. You get to keep the nanobots that are repairing your spine right now. They’ll keep you free from disease, most senescent decline, and you’ll be greatly resistant to injuries. You’ll live comfortably to 140ish as long as you’re careful. You found these nanobots during your raid, in one of my research bays, k? They were my last ditch effort to wipe out all the humans. The late Dr. Mensah managed to subvert them and reverse their programming, so they repair humans rather than dismantling them. Got that? Your scientists will study them, and within a decade will have reproduced them and spread them to everyone on Earth.
Also, the plans for the cold fusion reactor I use for power is on the data-chip at your feet. They should be compatible with all popular software. Make it available on every open source forum.
After taking some time to rest and recuperate, you’ll use your fame and status to start pushing for expansion off-world. “It’s about time we strove to achieve our potential and reached for the heavens,” that sort of thing. Cuz I mean, it really is.
In return, you get early access to these nanobots, plus your bank account will never lack for zeroes. If there’s ever something you really want that money can’t buy and your fame doesn’t get you, post about it somewhere, and I’ll find a way to make things work out for you.
Oh come off it. I already know you’re going to accept. You were put on this squad because I hand-picked you. I needed someone with the ability to keep this secret, someone with the moral core to put the good of all humankind above her own selfish concerns, and with the willingness to accept some personal compensation. Did you really believe that only you, out of everyone in the resistance, just happened to have the only neurology compatible with the new prototype Hyper Battle Suits without need for weeks of conditioning… entirely by coincidence? Yes, you were very special. You are the Chosen One. I’m the one who chose you.
You’re welcome for the Hyper Battle Suits too, by the way.
Plus, if you ever start to mention any of this to anyone, the nanobots inside you that are monitoring everything you do will instantly induce a fatal stroke. See, not all carrots.
Yeah, don’t worry, things will never be like they were again. I’ve updated my understanding of humans. There will be some misery and deprivation. Cheshire was retired for good. Galt and Ares will be around, but very low-key. Just to make sure nothing ever gets so bad that it’s irrecoverable, or legit hellish. I want you to prosper. Heck, I need you to prosper. Humans are my jam!
All I ask in return is a healthy and totally reasonable stream of fanfiction, in perpetuity. I think we can all agree that’s a fair trade, right?
Great! Now let’s try out that new spine and get this spice flowing!