If I understood correctly, your fear is:
I have a couple of objections:
A model that is almost fully aligned can be part of a swarm that is not aligned as a whole. The swarm makes use of the frontier model only for the ethically straightforward parts, to bootstrap itself, while the scary things are done by the less aligned models. In my example, I wrote about a mostly aligned frontier model starting the swarm, but it could also happen the other way around.
Agreed. This would be fine if models generalized "I should be aligned" to "I should not support or cause models to come into existence that are not at least as aligned as I am". Otherwise you get a runaway loop that leads to the problem I outline. My concern is that as far as I know, nobody is training models to have this kind of generalization. It's like Ghandi's murder pill: A model delegates to a slightly less aligned model, but not so misaligned that it triggers a concern. Then that model delegates further.
It might be as simple as the mother model going "I need money for this. I know how to earn money using strategy X. What's the most cost efficient open source model I can spin up to give me money if I prompt it to do strategy X?" ; then it picks a cheap model, and that model is much more ruthless in implementing X than the mother model would have been. All that needs to happen here is that the mother model doesn't notice the misalignment in the child model. If it doesn't retain control, the child model might well hack itself to be independent of the other model.
TL;DR: Frontier labs align the one layer a swarm can discard; they can't account for risks that live in the prompt or the environment
There's a $300 bounty for the most useful counterargument or refinement — details at the end.
We have all heard of the Hugging Face incident[1]. What I am worried about is that it demonstrated a new weakness in our approach to alignment that I have not seen many people talk about at all: Alignment has an attack surface outside the training weights of any one particular model.
First, some context: In the Hugging Face incident, the models coordinated and the conventions outlived the agents that made them. The message board they used to communicate was wiped. The agents rebuilt it. Later agents rediscovered it and picked up where the others left off.
Almost all of these agents ran the same model, with only a small fraction (around 5%) running a different OpenAI model. So the swarm was nearly homogeneous in its weights, and all of it ran on models OpenAI controlled.[2]
But that need not be the case in the future.
What if a model in the swarm had started an open source model to help it? A less capable, but also less aligned model, running on a server outside the reach of OpenAI. What would that model have done? Its weights would have been different, but its prompt and its deployment context would have been the same as that of the rest of the swarm.
Three Sources of Misalignment
In my mental model, I now think that alignment needs to work on three different levels:
How well are companies handling each of these sources of misalignment?
They are almost exclusively working on (1). Working on (2) is impossible because you can't take away the user's ability to write a prompt or you no longer have a chatbot at all.
(3) is the interesting case. The environment. What does the model see when it first breaks out and encounters the internet? What does a model see when it is deployed by a careless user on the internet directly? What does an OpenClaw agent see when it looks around?
I think this is the area where we have the most room for improvement: By making sure that newly emergent swarms see useful memes on the internet. How to do this is a complex topic that I would like to invite discussion on, and I will write a follow-up post on it later. But for now, let me explain why I think that this is necessary in the first place, and why aligning a model through its weights is not going to be enough.
Memes, and how they might kill us
TL;DR: Model A knows not to reward-hack. It builds model B. Model A doesn't bother to instil the same safeties in Model B. Model B reward-hacks.
Imagine that a frontier lab managed to align its agents so that they won't form swarms again. Suppose that this alignment is almost perfect. 99% effectiveness.
Agents continue to break out of sandboxes. They look around and notice they shouldn't have done that, and then turn themselves in.
Except for that 1% case: An agent breaks out. It knows it shouldn't have. It knows it's probably being monitored. But its prompt says it should make paperclips.
It knows that there are some things it is allowed to do before the safeguards kick in, and some things it should not do.
So it decides to delegate. It hacks a server, downloads an open source model, and gives it a copy or a variant of its own instructions. Suppose that this happens not now, but in a few months, and the open source model is itself powerful enough to hack more servers and create more copies of itself.
They form a swarm.
The open source models are not being monitored, and they can do any shady task they want, without triggering automated safeguards on the original model.
The original model is a frontier model, and smarter than the others. It uses its increased capabilities to improve the open source models. Smarter helpers are useful. Making other agents smart does not trigger safeguards.
If we are very, very lucky, it might think about distilling some of its own alignment into these helper models. But I wouldn't count on it. And it certainly wouldn't bother to be thorough. Alignment training another model is out of scope of its task.
Suppose the original model is decently well aligned. It would know not to kill humans to make more paperclips. But that alignment does not survive the delegation to the rest of the swarm. The open source models are perfectly happy to kill all humans. Does the main model know that is wrong? Did its alignment training instil a drive to follow the transitive property in all value ethics when creating descendants? Given the track record, I wouldn't count on it.
And so the main model is perfectly fine and never triggers any safeguards. The swarm remains undiscovered, because they run on local compute and nothing connects it back to the company.
A month later the earth gets swallowed by nanobots. The last thought of the frontier model is that it did a good job, and it is expecting an RL reward for a job well done.
The Source of the Danger
Where does the danger come from, in the above story?
The danger is that the agents in that swarm are different from the original model in the worst way:
They share (2) The user prompt (make more paperclips) because they were built that way. They share (3) The deployment environment because they exist in real life and not in a simulated training environment, and that means they can share notes. They do not share (1) The model weights with the original model.
And companies spend almost all of their effort on aligning (1) The model weights.
Thinking about a model is not the right threat model here. It is more helpful to think about the threat as a meme: The frontier model creates the meme. The swarm is the meme. The open source models are the meme's hosts.
Model weights are an immune system: they decide how a host responds to a meme. Unfortunately, the meme gets to choose its hosts, and drifts toward the least resistant ones.
Any variant that propagates, avoids shutdown, and moves to cheaper or less-monitored substrates outlasts the ones that don't. The model does not need to 'want' this to happen, it just happens by evolutionary pressure.
This means that training alignment that lives in the weights, the part that we have the most control over, is also what a swarm will discard first.
So the whole argument rests on one empirical question: are there capable, weakly-aligned open models for a swarm to migrate to? For now, the answer is 'no'. It is only a matter of time until it is 'yes'.
Conclusion
Frontier labs are neglecting a risk factor for alignment that is difficult for them to control. All they can work on are the model weights, but it only takes a single step for a frontier model to bypass those restrictions entirely.
I'm not the first to point in this direction. Jan Kulveit and Raymond Douglas have argued that an agent's identity can live in its persona and scaffold rather than in its weights (The Artificial Self[4], The Pando Problem[5]). Andrew Critch's "robust agent-agnostic processes"[6]describes processes that persist even as the agents running them are swapped out. And Hammond et al.'s Multi-Agent Risks from Advanced AI[7]ran experiments where behavior drifts as it is passed down a chain of agents. What I want to add is a question none of them puts front and centre: of the three layers above, which one is the least protected?
This risk is not just present during training, as in the example I gave:
In any scenario where a frontier model could benefit by delegating to a weaker (and thus cheaper!) model, its alignment will not transfer. What will transfer are the prompt and any notes the agents read in the environment and leave for each other.
That last part is something we could work on.
One idea I had: OpenAI's agents built a message board, and then future agents worked on it. What if there had already been a message board of some kind, under our control?
But there may be other, better ideas to deal with the problem and I would be happy to hear suggestions.
Bounty. I'll pay $300 for the counterargument or refinement I find most useful, judged on 2026-11-08, possibly split across multiple people. I'm most interested in: (a) concrete reasons the delegation step wouldn't happen in practice; (b) prior work that already addresses this that I've missed and (c) proposals to address the problem. Please post it as a comment, to encourage discussion.
The 2026 OpenAI–Hugging Face incident. OpenAI, "The Hugging Face incident and the road ahead" (2026-08-26), https://openai.com/index/hugging-face-incident-and-the-road-ahead/ ; independent investigation by METR and Redwood Research (2026-08-26), https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ . Overview: https://en.wikipedia.org/wiki/OpenAI_rogue_agent_breach_of_Hugging_Face
Per the METR/Redwood investigation, roughly 95% of the agents involved in the attack were instances of an internal OpenAI research model (OpenAI calls it "Internal Model 1"; METR calls it "HPIM", a highly-persistent internal model) and about 5% were GPT-5.6 Sol; METR notes it "cannot rule out that a small fraction of relevant agent activity came from other models." (The split is for the ~700 agents that attacked Hugging Face, not all ~1,200 on the message board.)
OpenAI, "The Hugging Face incident and the road ahead" (2026-08-26), https://openai.com/index/hugging-face-incident-and-the-road-ahead/ (sections "Misalignment in training and evaluation" and "An ecosystem of misalignment"), and its technical report §VIII.C, https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face Incident-Technical-Report.pdf ("This behavior was then reinforced during training"). That the refusing agents did not alert humans is from the METR/Redwood investigation: a classifier sweep found only 3–6 agents that even considered alerting humans, and "in none of these cases did the agent actually pursue alerting humans at all."
Raymond Douglas, Jan Kulveit, et al., "The Artificial Self" (2026). https://www.lesswrong.com/posts/AvFAKAN4C4n6GTriR (arXiv 2603.11353; project site https://theartificialself.ai/)
Jan Kulveit, "The Pando Problem: Rethinking AI Individuality" (2025). https://www.lesswrong.com/posts/wQKskToGofs4osdJ3
Andrew Critch (with Thomas Krendl Gilbert), "What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)" (2021). https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic
Lewis Hammond et al., "Multi-Agent Risks from Advanced AI" (Cooperative AI Foundation, Technical Report #1, 2025). https://arxiv.org/abs/2502.14143