I remember that example!
(EDITED: Crap. I included the key pic from the key research in the first draft of this comment, and then the site's software put my comment "close to the OP, with the OP not rolled out" and so the OP would be spoiled by the comment? Apologies. Hopefully with this edit to remove the pic from the comment things work better?)
This series is great! Please keep going!
It reminds me of some of the "best bits" from Accelerando, back in the day, when Stross was clearly alluding to ideas from actual published work by people trying to understand brains and learning algorithms and so on. Some parts of the novel were practically "a new allusion every paragraph" and the density of them caused me to laugh, which I take to be a positive some of like... uh... something interesting and probably good?
Also, more seriously, it is plausible that this very text might end up in GPT-N's language models. If the language models have a coherent literary concept of short sweet narratives describing obvious failure modes, that might be helpful? Like... in the actual future there will probably be engineers who re-use libraries very quickly to hit deadlines with high enough quality that the QA team can't instantly detect the problems. That "hurried productive iterative pragmatic chaos"-feeling is in these stories, and feels true to life.
Thank you for the kind words and the image. I tried and failed to locate the image while writing this story. I'm glad readers can see it now. I tried embedding your image under a spoiler tag but the spoiler tag only blocks text. Images go right through it. The link will have to do.
I didn't know Stross based his early work on published science. The earliest of his stories I read was Saturn's Children which seemed to be an allusion to Asimov's robot stories and Heinlein's Friday.
I mean... like... "the Lobsters" in Accelerando (who ultimately end up being highly similar to humans compared to the very weird "Vile Offspring", and altruistically are able to help us a bit with escaping, after humans are essentially deported from Earth and so on) were, I'm pretty sure, based on a hypothetical extension of work vaguely like this? (There's a specific study where someone did the crude but serviceable first cell of a Moravec Upload, but on a Lobster, that I wanted to link to but can't instantly find anymore.)
I can't remember all the allusions, but parts of that novel were very dense with these kinds of things :-)
My understanding is that Stross held (still holds (dunno: haven't checked in a while)) singularitarians in mild contempt, and just harvested a bunch of "our memes" and threw them back at us in a book aiming to insultingly pander to us. If that's what happened, I'm fine with it! Such high quality pandering is ok by me <3
I think over time he came up with larger audiences to insultingly pander to, where I could see the pandering more easily, but not notice as many collected allusions to interesting research? C'est la vie.
That may be one of the worst industrial definition accidents yet. Tensely awaiting the philosophical safety board report.
"No. They are just minimizing loss of capital," said Miriam.
Yikes, but in a country of how many billion at this point in the future(?) people are plentiful.
And this factory must have still been prototyping, otherwise I think there would be more dead bodies. Maybe if they decided to skip prototyping and move straight into production, at something like 500 units an hour, hitting the start button made things go really bad, really quick.
Also, Puppy Muffins. Retch.
"Nothing has entered of left the building since our arrival."
(I'm new on this site. How do I quote like Pattern did?)
Did you mean "...or left the building..."?
A green humvee arrived at Jieyang Chaoshan International Airport. Vi got in the back with Molly Miriam who handed her clipboard to Vi.
"健重制造公司. A no-name Chinese factory that makes barbells and similar equipment. It's not even fully-automated," read Vi.
"They are registered to use a low-intelligence AGI," said Miriam.
"What are we even doing here? Neither the product nor the AI poses a threat to civilization," said Vi.
"Something must have gone badly wrong," said Miriam.
The road to the factory was blockaded by the People's Liberation Army (PLA). The soldier at the checkpoint scanned the Bayeswatch agents' badges. A young officer—barely out of high school—escorted them inside the perimeter to Colonel Qiang.
"We could probably handle this on our own," said Colonel Qiang, "But protocol is protocol."
"So it is," said Miriam.
There were no lights on in the factory. No sound emanated from it. Fifty soldiers behind sandbags surrounded the factory, along with two armored personnel carriers and a spider tank.
"The police responded first. The sent a SWAT team in. Nobody came back. Then we were called. We would like to just burn the whole thing down. But this incident could be part of a wider threat. We cut power and Internet. Nothing has entered or left the building since our arrival. Rogue AIs can be unpredictable. We wanted your assessment of the situation before continuing," said Colonel Qiang.
"You did the right thing. This is probably an isolated incident. If so then the best solution is to rescue who we can and then level the building. Unfortunately, there is a chance this is not an isolated incident. Therefore our top priority is to recover the AI's hard drives for analysis," said Miriam.
"We will assault the building," said Colonel Qiang.
"You may have our sanction in writing. Assume humans are friendly and robots are hostile," said Miriam.
"Yes sir," said Colonel Qiang.
Miriam and Vi were quartered in a nearby building that had been comandeered by the PLA. They watched the assault from a video monitor.
"In training they taught us to never go full cyber against an AI threat," said Vi.
"That is correct," said Miriam.
"Which is why every assault force is no less than ten percent biological," said Vi.
Miriam nodded.
"Standard operating procedure is you go ninety percent robotic to minimize loss of life," said Vi.
"Ninety percent robotic does tend to minimize loss of life without the failure modes you get from going full cyber," said Miriam.
"It looks to me like they're going one hundred percent biological while their battle droids stay outside. Are we facing that dangerous of a hacking threat?" said Vi.
"No. They are just minimizing loss of capital," said Miriam.
The video feed of the factory was replaced by Colonel Qiang's face. "We have a survivor," he said.
Two privates guarded the door to their freely-liberated prisoner. His dress shirt was stained with blood and grease. An empty styrofoam take-out food tray lay in the corner of the table with a pair of disposable chopsticks and an empty paper cup. Miriam and Vi took seats opposite him.
"I understand you helped program the improperly registered AI at 健重制造公司," said Vi.
"I didn't know it was improperly registered," said Paul while looking straight at the security camera.
"We're not here to find out what laws were or weren't broken. We just want to know why there is a company of infantry surrounding this factory," said Miriam.
"There wasn't much to it. The mainframe running the assembly line barely qualifies as an AGI. We could never afford that much compute," said Paul.
"How does it work?" said Miriam.
"Labor is affordable here by international standards. Our factory is mostly human-run. Androids are expensive. We only have a couple of them. We should have been able to overpower robots if they were all that had gone rogue," said Paul.
"But that's not what happened," said Miriam.
"We didn't smell anything. People just started dying. We tried to help. More died. We tried to escape but the fire doors had been locked. I ran to my office, barricaded the door and breathed out the window," said Paul.
"Argon gas. It has all sorts of industrial applications," said Vi.
"Exactly," said Paul.
"And the same mainframe which controlled the robots also controlled the fire suppression system," said Vi.
Paul nodded.
"So why did it want to kill people?" said Vi.
"Maybe it was jealous," said Paul.
"Let's stick to the facts. Why use an AI at all if human labor is so cheap?" said Miriam.
"Human manual labor is cheap. New products are high margin but top designers are expensive. We had the AI do some manufacturing because embodiment helps with designing human-compatible products. But mostly we just used it for the generative model," said Paul.
Miriam flinched. "Thank you. That will be all," she said.
They were back in the monitor room.
"We don't need the hard drives. Do whatever you want," said Miriam to the image of Colonel Qiang.
The monitor went black.
"I lost count of how many OSHA regulations they violated," said Vi.
"OSHA has no jurisdiction here," said Miriam.
"Do you know what happened?" said Vi.
"When I was your age, I inspected a muffin factory. They followed all the regulations. It was even Three Laws Compliant. Very high tech. For its time," said Miriam.
Miriam lit a cigarette.
"The told the AI to make the cute muffins. They fed /r/cute into it as training data."
Miriam took a deep breath from her cigarette.
"The AI bought puppies. The engineers thought it was cute. They thought maybe they had solved the alignment problem," said Miriam.
Miriam took another swig. She exhaled slowly.
"The engineers had told the AI to make blueberry muffins. Do an image search for 'puppy muffin' on your phone," said Miriam.
"They do look the same. Puppies do look like blueberry muffins," said Vi.
"Puppy heads. Puppy heads look like blueberry muffins," said Miriam.
"Oh," said Vi.
"Come outside. You need to see this with your eyes," said Miriam.
The soldiers had retrieved several bodies. Doctors were autopsying them. The bodies' hands were missing. A few were missing half their forearms. One body had its neck and shoulders removed.
"They used a generative model on weightlifting equipment. They fed it pictures of people lifting weights. They annotated which regions of the images constituted a 'barbell'." said Miriam.
Vi almost puked.
"Tell me what happened," said Miriam.
"The generative model added disembodied hands to the barbell," said Vi.
Colonel Qiang ordered sappers to implode the facility.