1. Who are the customers actually buying all these products so that the auto-corporations can profit? They cannot keep their soulless economy going without someone to sell to, and if it's other AIs, why are those AIs buying when they can't actually use the products themselves?
2. What happened to the largest industry in developed countries, the service industry, which fundamentally relies on having an actual sophont customer to serve? (And again, if it's AIs, who the hell created AIs that exist solely to receive services they cannot actually enjoy, and how did they make money by doing that?)
3. Why didn't shareholders divest from auto-corporations upon realizing that they were likely to lead to ruin? (Don't say "they didn't realize it till too late", because you, right now, know it's a bad idea, and you don't even have money on the line.)
I ask these because, to be honest, I think this scenario is extremely far-fetched and unlikely. The worst thing that would happen if auto-corporations become a thing, in my mental model, is that existing economic inequalities would be permanently exacerbated due to the insane wealth accrued by their shareholders - because the only currently likely route to AGI, large language models, already understand what we actually mean when we say to maximize shareholder value, and won't paperclip-maximize, because they're not stupid, and they use language the same way humans do.
These are good questions!
I agree that this exact scenario is unlikely, but I think this class of failure mode is quite plausible, for reasons I hope I've managed to spell out more directly above.
Note that all of this relies on the assumption that we get AIs of a particular power level, and of a particular goodharting level, and a particular agency/coherency level. The AIs controlling future Earth are not wildly superhuman, are plausibly not particularly coherent in their preferences and do not have goals that stretch beyond Earth, no single system is a singleton, and the level of goodharting is just enough that humans go extinct but not so extreme that nothing humanly-recognisable still exists (though the Blight implies that elsewhere in the story universe there are AI systems that differ in at least some of these). I agree it is not at all clear whether these are true assumptions. However, it's not obvious to me that LLMs (and in particular AIs using LLMs as subcomponents in some larger setup that encourages agentic behaviour) are not on track towards this. Also note that a lot of the actual language that many of the individual AIs see is actually quite normal and sensible, even if the physical world has been totally transformed. In general, LLMs being able to use language about maximising shareholder value exactly right (and even including social responsibility as part of it) does not seem like strong evidence for LLM-derived systems not choosing actions with radical bad consequences for the physical world.
This is the story I use to express what a world where we fail looks like to left-leaning people who are allergic to the idea that AI could be powerful. It doesn't get the point across great, due to a number of things that continue to be fnords for left leaning folks which this story uses, but it works better than most other options. It also doesn't seem too far off what I expect to be the default failure case; though the factories being made of low-intelligence robotic operators seems unrealistic to me.
I opened it now to make this exact point.
Thanks for the review! Curious what you think the specific fnords are - the fact that it's very space-y?
What do you expect the factories to look like? I think an underlying assumption in this story is that tech progress came to a stop on this world (presumably otherwise it would be way weirder, and eventually spread to space).
It's striking that there are so few concrete fictional descriptions of realistic AI catastrophe, despite the large amount of fiction in the LessWrong canon. The few exceptions, like Gwern's here or Gabe's here, are about fast take-offs and direct takeover.
I think this is a shame. The concreteness and specificity of fiction make it great for imagining futures, and its emotional pull can help us make sense of the very strange world we seem to be heading towards. And slower catastrophes, like Christiano's What failure looks like, are a large fraction of a lot of people's p(doom), despite being less cinematic.
One thing that motivated me in writing this was that Bostrom's phrase "a Disneyland without children" seemed incredibly poetic. On first glance it's hard to tell a compelling or concrete story about gradual goodharting: "and lo, many actors continued to be compelled by local incentives towards collective loss of control ..."—zzzzz ... But imagine a technological and economic wonderland rising, but gradually disfiguring itself as it does so, until you have an edifice of limitless but perverted plenty standing crystalline against the backdrop of a grey dead world—now that is a poetic tragedy. And that's what I tried to put on paper here.
Did it work? Unclear. On the literary level, I've had people tell me they liked it a lot. I'm decently happy with it, though I think I should've cut it down in length a bit more.
On the worldbuilding, I appreciated being questioned on the economic mechanics in the comments, and I think my exploration of this in the comments is a decent stab at what I think is a neglected set of questions about how much the current economy being fundamentally grounded in humans limits the scope of economic-goodharting catastrophes. Recently, I discovered earlier exploration of very similar questions in Scott Alexander's 2016 "Ascended economy?", and by Andrew Critch here. I also greatly appreciated Andrew Critch's recent (2024) post raising very similar concerns about "extinction by industrial dehumanization".
I continue to hope that more people work on this, and that this piece can help by concretising this class of risks in people's minds (I think it is very hard to get people to grok a future scenario and care about it unless there is some evocative description of it!).
I'd also hope there was some way to distribute this story more broadly than just on LessWrong and my personal blog. Ted Chiang and the Arrival movie got lots of people exposed to the principle of least action—no small feat. It's time for the perception of AI risk to break out of decades of Terminator comparisons, and move towards a basket of good fictional examples that memorably demonstrate subtle concepts.
I enjoyed this but I didn't understand the choice of personality for Alice and Charlie, it felt distracting. I would have liked A&C to have figured out why this particular Blight didn't go multi-system.
I really appreciate that this has a link besides to this site, so I can link it to those who find the lesswrong link to be an instant turnoff and instead of opening the link, criticize me for ever browsing lesswrong. Many of them are the ones who most need to hear this.
The spaceship swung into orbit around the blue-grey planet with a final burn of its engines. Compared to the distance they had travelled, the world, now only some four hundred kilometres below and filling up one hemisphere of the sky, was practically within reach. But Alice was no less confused.
"Well?" she asked.
Charlie stared thoughtfully at the world slowly rotating underneath their feet, oceans glinting in the sunlight. "It looks lickable", he said.
"We have a task", Alice said, trying to sound gentle. Spaceflight was hard. Organic life was not designed for it. But their mission was critical, they needed to move fast, and Charlie, for all his quirks, would need to be focused.
"What's a few minutes when it will take years for anything we discover to be known back home?" Charlie asked.
"No licking", Alice said.
Charlie rolled his eyes, then refocused them on the surface of the planet below. They were just crossing the coast of one of the larger continents. Blue water was giving way to grey land.
"Look at the texture", Charlie said. They had seen it from far away with telescopes, but there was something different about seeing it with their bare eyes. Most of the land surface of the planet was like a rug of fine grey mesh. If there had been lights, Alice would have guessed the entire planet's land was one sprawling city, but as far as their instruments could tell, the world had no artificial lighting.
As far as they could tell, the world also had no radio. They had broadcast messages at every frequency they could, and in desperation even by using their engines to flash a message during their deceleration burn. No response had come.
Alice pulled up one of the telescope feeds on the computer to look closer at the surface. She saw grey rectangular slabs, typically several hundred metres on a side, with wide roads running between them. The pattern was not perfect - sometimes it was irregular, and sometimes there were smaller features too. Some of the smaller ones moved.
"Are they factories?" Charlie asked.
"I'd guess so", Alice said, watching on the telescope feed as a steady stream of rectangular moving objects, each about ten metres long, slid along a street. Another such stream was moving along an intersecting street, and it looked like they would crash at the intersection, but the timing and spacing was such that vehicles from one stream crossed the road just as there were gaps in vehicles along the other stream.
"A planet covered by factories, then", Charlie said. "With no one home to turn the lights on."
"I want to see what they're making", Alice said.
--
All through the atmospheric entry of their first drone package, Alice sat tight in her seat and clenched and unclenched her hands. So far all they had done was passive observation or broadcasting. A chunky piece of hardware tracing a streak of red-hot plasma behind it was a much louder knock. She imagined alien jet fighters scrambling to destroy their drones, and some space defence mechanism activating to burn their ship.
The image she saw was a jittery camera feed, showing the black back of the heatshield, the grey skin of the drone package, and a sliver of blue sky. It shook violently as the two halves of the heatshield detached from each other and then the drone package, tumbling off in opposite directions. Land became visible, kilometres below, the grey blocks of the buildings tiny like children's blocks but still visibly three-dimensional, casting shadows and moving as the drone package continued falling.
The three drones tested their engines, and for a moment flew - or at least slowed their descent - in an ungainly joint configuration, before breaking off from each other and spreading their wings to the fullest. The feed showed the other two drones veering off into the distance on wide narrow wings, and then the view pulled up as the nose of the drone lifted from near-vertical to horizontal.
"Oops, looks like we have company", Charlie said. He had been tapping away at some other screens while Alice watched the drone deployment sequence.
Alice jumped up from her seat. "What?"
"Our company is ... a self-referential joke!"
Alice resisted the temptation to say anything and instead sunk back into her seat. On her monitor, the grey blocks continued slowly moving below the drone. She tapped her foot against the ground.
"Actually though", Charlie said. "We're not the only ones in orbit around this planet."
"What else is orbiting? Has your sense of shame finally caught up with you and joined us?"
"Looks like satellites. Far above us, though. Can you guess how far?"
"I'd guess approximately the distance between you and maturity, so ... five light-years?"
Charlie ignored her. "Exactly geostationary altitude", he said, grinning. The grin was like some platonic ideal of intellectual excitement; too pure for Alice's annoyance to stay with her, or for her to feel scared about the implications.
"But nothing in lower orbits?" Alice asked.
"No", Charlie said. "Someone clearly put them there; stuff doesn't end up at exactly geostationary altitude unless someone deliberately flies a communications or GPS satellite there. Now I can't be entirely sure that the geostationary satellites are completely dead, but I'd guess that they are."
"Like everything else", Alice said, but even as she said so she caught sight of a long trail of vehicles making its way along one of the roads. There was something more real about seeing them on the drone feed.
"Maybe this is just a mining outpost", Charlie said. "Big rocket launch to blast out a billion tons of ore to god-knows-where, once a year."
"Or maybe they're hiding underground or in the oceans", Alice said.
"Let's get one of the drones to drop a probe into the oceans. I'll send one of our initial trio over to the nearest one, it's only a few hundred kilometres away", Charlie said.
"Sure", Alice said.
They split the work of flying the drones, two of them mapping out more and more of the Great Grey Grid (as Alice took to calling it in her head), and one flying over the planet's largest ocean.
Even the oceans were mostly a barren grey waste. Not empty, though. They did eventually see a few small scaly fish-like creatures that stared at their environment with uncomprehending eyes. Alien life. A young Alice would have been ecstatic. But now she was on a mission, and her inability to figure out what had happened on this planet annoyed her.
In addition to the ocean probe, they had rovers they could send crawling along the ground. Sometimes the doors of the square buildings were open, and Alice would drive a rover past one opening. Most seemed to either be warehouses of stacked crates, or then there would be some kind of automated assembly line of skeletal grey robot arms and moving conveyor belts. A few seemed to place more barriers between the open air and their contents; what went on there, the rovers did not see.
The first time Alice tried to steer a rover into a building, it got run over by a departing convoy of vehicles. The vehicles were rectangular in shape but with an aerodynamic head, with three wheels on each side. Based on their dimensions, she could easily imagine one weighing ten or twenty tons. The rover had no chance.
"Finally!" Charlie had said. "We get to fight these aliens."
But there was no fight. It seemed like it had been a pure accident, without any hint of malice. The grey vehicles moved and stopped on some schedule of their own, and for all Alice knew they were not just insensitive beasts but blind and dumb ones too.
The next rover got in, quickly scooting through the side of the entrance and then off to one side, out of path of the grey vehicles. It wandered the building on its own, headlights turned on in the otherwise-dark building to bring back a video stream of an assembly line brooded over by those same skeletal hands they had glimpsed from outside. Black plastic beads came in by the million on the grey vehicles. A small thin arm with a spike on the end punctured a few holes on one side, and using these holes two of the black beads were sown onto an amorphous plushy shape. The shape got appendages, were covered with a layer of fluff, and the entire thing became a cheerful purple when it passed through an opaque box with pipes leading into it. It looked like a child's impression of a hairy four-legged creature with black beady eyes above a long snout. A toy, but for who?
The conveyor belt took an endless line of those fake creatures past the rover's camera at the end of the assembly line. Alice watched them go, one by one, and fall onto the open back of a grey vehicle. It felt like each and every one made eye contact with her, beady black eyes glinting in the light. She watched for a long time as the vehicle filled up. Once it did, a panel slid over the open top to close the cargo bay, and it sped off out the door. The conveyor belt kept running, but there was a gap of a few metres to the next plushy toy. It came closer and closer to the end - and suddenly a vehicle was driving into place, and the next creature was falling, and it just barely fell into the storage hold of the vehicle while it was driving into place.
"How scary do you find the Blight?" Alice asked.
"Scary enough that I volunteered for this mission", Charlie said.
Alice remembered the charts they had been shown. They had been hard to miss; even the news, usually full of celebrity gossip and political machinations, had quickly switched to concentrating on the weirdness in the sky once the astronomers spotted it. Starlight dimming in many star systems and what remained of the the light spectra shifting towards the infrared. Draw a barrier around the affected area, and you get a sphere 30 light-years wide, expanding at a third of the speed of light. At the epicentre, a world that had shown all the signs of intelligent life that could be detected from hundreds of light-years away - a world that astronomers had broadcast signals to in the hopes of finally making contact with another civilisation - that had suddenly gone quiet and experienced a total loss of oxygen in its atmosphere. The Blight, they had called it.
In the following years, civilisation had mobilised. A hundred projects had sprung forth. One of them: go investigate the star system that was the second-best candidate for intelligent life, but had refused to answer radio signals, and see if someone was there to help. That was why they were here.
"I think I found something as scary as the Blight", Alice said. "Come look at this."
The purple creatures kept parading past the camera feed
--
Over the next five days, while the Blight advanced another forty billion kilometres towards everything they loved back home, Alice and Charlie were busy compiling a shopping catalogue.
"Computers", Alice said. "Of every kind. A hundred varieties of phones, tablets, laptops, smartwatches, smartglasses, smart-everything."
"Diamonds and what seems to be jewellery", Charlie said.
"Millions of tons of every ore and mineral." They had used their telescopes on what seemed to be a big mine, but they had barely needed them. It was like a huge gash in the flesh of a grey-fleshed and grey-blooded giant, complete with roads that looked like sutures. There were white spots in the image, tiny compared to the mine, each one a sizeable cloud.
"Clothes", Charlie continued. "Lots and lots of clothes of different varieties. They seem to be shipped around warehouses until they're recycled."
"Cars. Sleek electric cars by the million. But we never see them used on the roads, though there are huge buildings were brand-new cars are recycled. And airplanes, including supersonic ones."
"A lot of things that look like server farms", Charlie said. "Including ones underwater and on the poles. There's an enormous amount of compute in this world. Like, mind-boggling. I was thinking we should figure out how to plug into all of it and mine some crypt-"
"Ships with nuclear fusion reactors", Alice interrupted. There were steady trails of them cutting shortest-path routes between points on the coast.
"Solar panels", Charlie said. "Basically every spare surface. The building roofs are all covered with solar panels."
"And children's plush toys", Alice said.
They were silent for a while.
"We have a decent idea of what these aliens looked like", Alice said. "They were organic carbon-based lifeforms, like us. Similar in size too, also bipedal. And it's like they left some ghostly satanic industrial amusement park running, going through all the motions in their absence, and disappeared."
"And they didn't go to space, as far as we know", Charlie said.
"At least we don't have any more Blights to worry about then", Alice said. "I can't help but imagining that the Blight is something like this. Something that just tiles planets with a Great Grey Grid, does something even worse to the stars, and then moves on."
"They had space technology, but apparently whoever built the Great Grey Grid didn't fancy it", Charlie said. "The satellites might predate it. Probably there were satellites in lower orbits too, but their orbits decayed and they fell down, so we only see the geostationary ones up high."
"And then what?" Alice said. "All of them vanished into thin air and left behind a highly-automated ghost-town?"
Charlie shrugged.
"Can we plug ourselves into their computers?" Alice asked.
"To mine cr-?"
"To see if anyone's talking."
Charlie groaned. "You can't just plug yourself into a communication system and see anything except encrypted random-looking noise."
"How do you know they encrypt anything?"
"It would be stupid not to", Charlie said.
"It would be stupid to blind yourself to the rest of the universe and manufacture a billion plush toys", Alice said.
"Seems like it will work for them until the Blight arrives."
--
Alice floated in the middle of the central corridor of the ship. The ship was called Legacy, but even before launch they had taken to calling it "Leggy" for short. The central corridor linked the workstation at the front of the ship where they spent most of their days to the storage bay at the back. In the middle of the corridor, three doors at 120-degree angles from each other lead to the small sleeping rooms, each of them little more than a closet.
Alice had woken up only a few minutes ago, and still felt an early-morning grogginess as well as the pull of her bed. The corridor had no windows or video feeds, but was dimly lit by the artificial blue light from the workstation. They were currently on the night side of the planet.
She took a moment to look at the door of the third sleeping room. It was closed, like always, with its intended inhabitant wrapped in an air-tight seal of plastic in a closed compartment of the storage bay. They would flush him into space before they left for home again; they could have no excess mass on the ship for the return journey.
Alice thought again of the hectic preparations for the mission. Apart from Blightsource, this was only one planet the astronomers had spotted that might have intelligent life on it, and the indications were vague. But when you look into space and see something that looks like an approaching wall of death - well, that has a certain way of inspiring long-shots. Hence the mission, hence Legacy's flight, hence crossing over the vast cold stretch of interstellar space to see if any answers could be found on this world. Hence Bob's death while in cryonic suspension for the trip. Hence the hopes of all civilisation potentially resting on her and Charlie figuring valuable out something.
If Charlie and she could find something on this world, some piece of insight or some tool or weapon among the countless pieces of technological wizardry that this world had in spades, that had a credible chance against the Blight when it arrived ... maybe there was hope.
Alice pushed off on the wall and set herself in a slow spinning motion. The ship seemed to revolve around her. Bob's door revolved out of sight, and Charlie's door became visible -
Wait.
Her gravity-bound instincts kicked in and she tried to stop the spin by shoving back with her hands, but there was nothing below her, so she remained spinning slowly. She breathed in deeply to calm herself down, then kicked out a foot against the wall to push herself to the opposite one. She grabbed one of the handles on the wall and held onto it.
The light on Charlie's room was off. That meant it was empty.
"Charlie!" Alice called.
No response.
The fear came fast. Here she was, light-years from home, perhaps all alone on a spaceship tracing tight circles around a ghostly automated graveyard planet. The entire mass of the planet stood between her and the sun. Out between the stars, the Blight was closing in on her homeworld. She counted to calm herself down; one, two, three, ... and just like that, the Blight was three hundred thousand kilometres closer to home. Unbidden, an image of the fluffy purple creature popped up in her mind, complete with its silly face and unblinking eye contact.
Soundlessly, she used the handles on the wall of the corridor to pull herself towards the workstation. She reached the door, peered inside -
There was Charlie, staring at a computer screen. He looked up and saw Alice. "You scared me!" he said. "Watch out, no need to sneak behind me so quietly."
"I called your name", Alice said.
"I know, I know", Charlie said. "But I'm on to something here, and I just want to run a few more checks and then surprise you with the result."
"What result?" Alice glanced at some of the screens. Two of the drones were above the Great Grey Grid, one above ocean. With their nuclear power source, they could stay in the air as long as they wanted. Even though their focus was no longer aerial reconnaissance, there was no reason not to keep them mapping the planet from up close, occasionally picking up things that their surveys from the ship did not.
"I fixed the electrical issues with the rover and the cable near the data centre", Charlie said.
"So you're getting data, not just frying our equipment?"
"Yes", Charlie said. "And guess what?"
"What?"
"Guess!"
"You found a Blight-killer", Alice said.
"No! Even better! These idiots don't encrypt their data as far as I can tell. And I think a lot of it is natural language."
"Okay, and can we figure out what it means?"
"We have automated programs for trying to derive syntax rules and so on", Charlie said. "It's already found something, including good guesses of which words are prepositions and what type of grammar they have. But mapping words to meaning based on purely statistics of how often they occur is hard."
"I've seen products they have with pictures and instruction manuals", Alice said. "We could start there."
"Oh no", Charlie said. "This is going to be a long process."
--
By chance, it turned out not to be. Over the next day, they had sent a rover to a furniture factory and had managed, after some attempts, to steal an instruction leaflet out of a printer before the robotic arm could snatch it to be packaged with the furniture. Somehow Alice was reminded of her childhood adventures stealing fruit from the neighbour's garden.
They had figured out which words meant "cupboard", "hammer", and "nail", and so on. But then another rover on the other side of the world had seen something. It was exploring a grey and windy coast. On one side of the rover was the Great Grey Grid and the last road near the coast, the occasional vehicle hurtling down it. But on the other side was a stretch of rocky beach hammered by white-tipped waves, a small sliver of land that hadn't been converted to grey.
The land rose by the beach, forming a small hill with jagged rocky sides. The sun shone down on one face of it, but there was a hollow, or perhaps small cave, that was left in the dark by the overhanging rock. And in the rock around this entrance, there were several unmistakable symbols scratched into the rock, each several metres high.
Alice took manual control of the rover and carefully instructed it to drive over the rocky beach towards the cave entrance. On the way it passed what seemed to be a fallen metal pole with some strips of fabric still clinging to it.
Once it was close enough to the mouth of what turned out to be a small cave, the camera could finally see inside.
There was a black cabinet inside. Not far from it, lying on the ground, was the skeleton of a creature with four slender limbs and a large head. Empty eye sockets stared out towards the sky.
Alice felt her heart beating fast. It wasn't quite right; many of the anatomical details were off. But it was close enough, the similarity almost uncanny. Here, hundreds of light years away, evolution had taken a similar path, and produced sapience. And then killed it off.
"Charlie", she said in a hoarse voice.
"What?" Charlie asked, sounding annoyed. He had been staring at an instruction manual for a chair, but he looked up and saw the video feed. "Oh", he said, in a small voice. "We found them."
Alice tore her eyes away from the skeleton and to the small black cabinet. It had a handle on it. She had the rover extend an arm and open it.
--
The capsule docked with Leggy and in the weightless environment they pushed the cabinet easily into the ship. They had only two there-and-back-again craft - getting back to orbit was hard - but they had quickly decided to use one to get this cabinet up. It had instructions, after all; very clear instructions, though ones that their rovers couldn't quite follow.
It started from a pictographic representation, etched onto plastic cards, of how you were supposed to read the disks. They managed to build something that could read the microscopic grooves on the disk as per the instructions, and transfer the data to their computers.
After a few hours of work, they had figured out the encodings for numbers, the alphabet, their system of units, and seemingly also some data formats, including for images.
Confirmation came next. The next item on the disk was an image of two of the living aliens, standing on a beach during a sunset. Alice stared into their faces for a long time.
Next there came images next to what were clearly words of text, about fifty of them. Some of the more abstract ones took a few guesses, but ultimately they thought they had a base vocabulary, and with the help of some linguistics software, it did not take very long before they had a translated vocabulary list of about eight thousand words.
Alice was checking the work when Charlie almost shouted: "Look at this!"
Alice looked at what he was pointing at. It was a fragment of text that read:
"What is this? Is Mr Skeleton trying to sell us furniture from beyond the grave?" Alice asked.
"No", Charlie said. "This isn't what I got from the recovered data; I haven't looked at the big remaining chunk yet. This is what I got by interpreting one of the packets of data running on the cables that our rover is plugged into using what we now know about their data formats and the language."
"And?"
"I don't get it!" Charlie said. "Why would a world of machines send each other emails in natural language?"
"Why would they manufacture plushy toys? I doubt the robotic arms need cuddles."
Charlie looked at the world, slowly spinning underneath their ship. "Being so close to it makes me feel creeped out. I don't get it."
"You don't want to lick it anymore?" Alice asked. She decided not to tell Charlie about her own very similar feelings earlier, when she thought for a moment Charlie had gone missing.
Charlie ignored her. "I think the last thing on Mr Skeleton's hard-drive is a video", he said. "I've checked and it seems to play."
"You looked at it first?" Alice said in a playfully mocking tone. The thrill of discovery was getting to her.
"Only the first five frames", Charlie said. "Do you want to watch it?"
--
Our Civilisation: A Story read a short fragment of subtitle, white on black, auto-translated by a program using the dictionary they had built up.
There was a brief shot of some semi-bipedal furry creature walking in the forest. Then one of a fossilised skeleton of something more bipedal and with a bigger head. Then stone tools: triangular ones that might have been spear tips, saw-toothed ones, clubs. A dash of fading red paint on a rock surface, in the shape of a cartoon version of that same bipedal body plan.
There were two pillars of stone in a desert on what looked like a pedestal, some faded inscription at its base and the lone and level sands stretching far away. There was a shot of an arrangement of rocks, some balancing on top of two others, amid a field of green. A massive pyramidal stone structure, lit by the rising sun.
Blocky written script etched on a stone tablet. Buildings framed by columns of marble. A marble statue of one of the aliens, a sling carelessly slung over its shoulder, immaculate in its detail. A spinning arrangement of supported balls orbiting a larger one. And still it moves, the subtitles flashed.
A collection of labelled geometric diagrams on faded yellow paper. Mathematical Principles of Natural Philosophy.
A great ornate building with a spire. A painting of a group of the aliens clad in colourful clothing. An ornate piece of writing. We hold these truths to be self-evident ...
A painting of a steam locomotive barrelling along tracks. A diagram of a machine. A black-and-white picture of one of the aliens, then another. Government of the people, for the people, by the people, shall not perish ...
An alien with white hair sticking up, holding a small stick of something white and with diagrams of cones behind him. Grainy footage of propeller aircraft streaking through the sky, and then of huge masses of people huddling together and walking across a barren landscape, and then of aliens all in the same clothes charging a field, some of them suddenly jerking about and falling to the ground. We will fight on the beaches, we will fight on the landing grounds ...
A black-and-white footage of a mushroom cloud slowly rising from a city below. A picture, in flat pale blue and white, showing a stylised representation of the world's continents. The same picture, this time black-and-white, on the wall of a room where at least a hundred aliens were sitting.
An alien giving a speech. I have a dream. An alien, looking chubby in a space suit, standing on a barren rocky surface below an ink-black sky next to a pole with a colourful rectangle attached to it.
Three aliens in a room, looking at the camera and holding up a piece of printed text. Disease eradicated.
What looked like a primitive computer. A laptop computer. An abstract helical structure of balls connected by rods, and then flickering letters dancing across the screen.
A blank screen, an arrow extending left to right across it - time, flashed the subtitles- and then another arrow from the bottom-left corner upwards - people in poverty - and then a line crawling from left to right, falling as it did so.
A line folding itself up into a complicated shape. AI system cracks unsolved biology problem.
From then on, the screen showed pictures of headlines.
All routine writing tasks now a solved problem, claims AI company.
Office jobs increasingly automated.
Three-fourths of chief executives of companies on the [no translation] admit to using AI to help write emails, one-third have had AI write a shareholder letter or strategy document.
Exclusive report: world's first fully-automated company, a website design agency.
Mass layoffs as latest version of [no translation] adopted at [no translation]; 'stunning performance' at office work.
Nations race to reap AI productivity gains: who will gain and who will lose?
CEO of [no translation] resigns, claiming job pointless, both internal and board pressure to defer to "excellently-performing" AI in all decisions.
[No translation] ousts executive and management team, announces layoffs; board supports replacing them with AI to keep up with competition.
Entirely or mostly automated companies now delivering 2.5x higher returns on investment on average; 'the efficiency difference is no joke', says chair of [no translation].
Year-on-year economic growth hits 21% among countries with advanced AI access.
Opinion: the new automated economy looks great on paper but is not serving the needs of real humans.
Mass protests after [no translation], a think-tank with the ear of the President, is discovered to be funded and powered by AI board of [no translation], and to have practically written national economic policy for the past two years.
'No choice but forward', says [no translation] after latest round of worries about AI; unprecedented economic growth still strong.
[No translation 1] orders raid of [no translation 2] over fears [no translation 2] is not complying with latest AI use regulations, but cannot execute order due to noncompliance from the largely-automated police force; 'we are working with our AI advisers and drivers in accordance with protocol, and wish to assure the [no translation 3] people that we are still far from the sci-fi scenario where our own police cars have rebelled against us.'
'AI overthrow' fears over-hyped, states joint panel of 30 top AI scientists and business-people along with leading AI advisory systems; 'they're doing a good job maximising all relevant metrics and we should let them keep at it, though businesses need to do a better job of selecting metrics and tough regulation is in order.'
Opinion: we're better-off under a regime of rigorous AI decision-making than under corrupt politicians; let the AIs repeat in politics what they've done for business over the last five years.
The statistics have never looked so good' - Prime Minister reassures populace as worries mount over radical construction projects initiated by top AI-powered companies.
Expert panel opinion: direct AI overthrow scenario remains distant threat, but more care should be exercised over choice of target metrics; recommend banning of profit-maximisation target metric.
Movement to ban profit-maximising AIs picks up pace.
Top companies successfully challenge new AI regulation package in court.
'The sliver of the economy over which we retain direct control will soon be vanishingly small', warns top economist, 'action on AI regulation may already be too late'.
Unverified reports of mass starvation in [no translation]; experts blame agricultural companies pivoting to more land-efficient industries.
Rant goes viral: 'It's crazy, man, we just have these office AIs that only exist in the cloud, writing these creepily-human emails to other office AIs, all overseen by yet another AI, and like most of their business is with other AI companies; they only talk to each other, they buy and sell from each other, they do anything as long as it makes those damned numbers on their spreadsheets just keep ticking up and up; I don't think literally any human has ever seen a single product out of the factory that just replaced our former neighbourhood, but those factories just keep going up everywhere.'
Revolution breaks out in [no translation]; government overthrown, but it's business-as-usual for most companies, as automated trains, trucks, and ships keep running.
[No translation] Revolution: Leaked AI-written email discovered, in which the AI CEO ordered reinforcement of train lines and trains three weeks ago. 'We are only trying to ensure the continued functioning of our supply chains despite the recent global unrest, in order to best serve our customers', CEO writes in new blog post.
[No translation] Revolution: crowds that tried swarming train lines run over by trains; 'the trains didn't even slow down', claim witnesses. CEO cites fiduciary duties.
Despite unprecedented levels of wealth and stability, you can't actually do much: new report finds people trying to move house, book flight or train tickets, or start a new job or company often find it difficult or impossible; companies prioritising serving 'more lucrative' AI customers and often shutting down human-facing services.
Expert report: 'no sign of human-like consciousness even in the most advanced AI systems', but 'abundantly clear' that 'the future belongs to them'.
New report: world population shrinking rapidly; food shortages, low birth rates, anti-natalist attitudes fuelled by corporate campaigns to blame.
The screen went blank. Then a video of an alien appeared, sitting up on a rocky surface. Alice took a moment to realise that it's the same cave they found the skeleton in. The alien's skin was wrapped tight around its bones, and even across the vast gulf of biology and evolutionary history, Alice could tell that it is not far from death. It opened its mouth, and sound came out. Captions appeared beneath it.
"It is the end", the alien said, its eyes staring at them from between long unkempt clumps of hair. "On paper, I am rich beyond all imagination. But I have no say in this new world. And I cannot find food. I will die."
The wind tugged at the alien's long hair, but otherwise the alien was so still that Alice wondered if it had died there and then.
"There is much I would like to say", the alien says. "But I do not have the words, and I do not have the energy." It paused. "I hope it was not all in vain. Or, that if for us it was, that for someone up there it isn't."
The video went blank.
Alice and Charlie watched the blank screen in silence.
"At least the blight they birthed seems to have stuck to their world", Charlie said after a while.
"Yeah", Alice said, slowly. "But I don't think we'll find anything here."
Legacy completed nine more orbits of the planet, and then jettisoned all unnecessary mass into space. Its engines jabbed against the darkness of space, bright enough to be visible from the planet's surface. There was no one to see them.
On a factory down on the planet, an assembly line of beady-eyed purple plush toys marched on endlessly.
---
The title of this work is taken from a passage in Superintelligence: Paths, Dangers, Strategies, where Nick Bostrom writes:
The outline of events presented draws inspiration from several sources, but most strongly on Paul Christiano's article What failure looks like.
---
Any advice on distributing this story more widely is welcome.