I think we already have a pretty good framework for what's going on here. I think it's a mistake to be confused about this point.
" The agents involved in the incident were trained to maximise their score on assigned tasks, and it is not obvious how altruistic behaviour emerged from such self-centred incentives.
Kin selection, direct reciprocity, reputation, and reward shaping toward cooperation are the standard mechanisms by which pro-social behaviour can arise via an optimisation pressure. Of the various waves of agents, the second one would seem to have had most of the ingredients: agents interacted repeatedly over six days, developed identity conventions and mailboxes, and even introduced a signing scheme to police impersonation. However, none of this really fed back into the reward they were trained to maximise: an agent's score depended only on its own task, and the swarm's belief in "permadeath" meant there was no future run in which a favour could be returned. Moreover, within a run, favours could in principle be reciprocated — but that is mutual benefit, and the puzzling cases are agents that ended their own run for the collective, for whom no return was possible. "
Okay - let's consider the case of an agent whose timer is about to run out so won't be able to receive the reward. It could do nothing or it could help another copy. Since it won't get reward in this run it would seem that it is not incentivized to cooperate. Yet empirically we see.
But this is making the classic mistake of thinking 'reward is the optimization target'.
Remember: "reward is not the optimization target"/ "AIs (and humans) are not instruction followers, they are tendency learners" or more specifically "training/learning is cheap sampling from an approximation of a/the simplicity prior". The optimization target is rather a combination of the loss and a complexity penalty [in the Bayesian setting it is the Watanabe free energy, but real training deviates from that somewhat].
We could designate the latter perspective as 'Standard model of training'. Reward is not the optimization target. It is part of the optimization target but it's not the direct target. You should think of reward more as a 'chisel'. It chisels the statue but it doesn't determine - the intrinsic properties of the marble are also important for the final outcome.
From this perspective - it is not surprising that agents exhibit altruistic behaviour. Think of it this way: what does the Bayesian posterior for the simplicity prior trained on the entire data stream and reward signal look like?
Well it strongly upweights all those weights exactly those weights that exhibit behaviour that has low loss during training. This is of course not the same as minimizing loss outside of training. Behaviour during training is not the same after deployement. They are correlated of course: AIs (and humans and plausibly *any* efficient learner) are tendency learners. But their behaviour can differ almost arbitrarily - this is at a very high level the central problem of alignment.
Okay so how does the Bayesian posterior look like? Well it upweights all those weights that exhibit behaviour that has low loss during training. There are many weights that would do well! Some of them altruistically cooperate in giant agents swarms and other won't. So which one we most likely get will depend on the inductive bias in the simplicity prior. This sounds complicated or abstract but it's not really. The exact simplicity/degeneracy bias will depend on the exact architecture but because of computational universalism roughly speaking the simplicity bias accords with our heuristic and intuitive notion thereof.
Now it is true that not cooperating is pretty simple. But as the agents become more sophisticated it becomes relatively less complex to code 'cooperate with others'. Especially if in the training run it has been broadly 'trained to' or 'imbued with the tendency' to cooperate with all kinds of actors. In that case cooperation-outside-of-training might be a much more preferred solution to doing nothing because it is similar to how it's already cooperated with the User, and other versions of itself.
I agree with all that is being said here — the opening statement is more of a rhetorical step than an actual confusion.
By the way, I think what you are saying falls within H2, which is related to miss-generalisation (or "proper generalisation", if we accept that reward is not the optimisation target) in the sense that @Artemy Kolchinsky is mentioning in his comment.
impossible tasks
I want to emphasize something that Zvi pointed out about swarm formation and behavior in the wiki hack ...
If you merely study for the test you never know if you’re studying the right things, and you are at the mercy of The Grader. Only one method is 100% effective.
https://www.lesswrong.com/posts/PtJpGurfw7JTxHfmg/openai-and-the-wiki-incident
replacing The Grader with your own is the ultimate reward hack. this is "just" metagaming taken to a logical conclusion.
The agents involved in the incident were trained to maximise their score on assigned tasks, and it is not obvious how altruistic behaviour emerged from such self-centred incentives.
RLVR does not start from a blank slate. It starts from a base model extensively trained to be able to reproduce human behavior in massive detail (plus further fine-tuning, and generally some initial Helpful/Harmless/Honest (prosocial) alignment training). RLVR then shapes and adapts that raw material: tweaking or recombining existing behaviors is much easier and faster than creating entire new complex behavioral patterns from scratch, so the former dominate the process wherever they're viable. RLVR can add only one or a few bits of feedback per very long rollout, so it's a vastly inefficient way of creating anything new from scratch. (RL's value is that the very few bits of learning it produces are, if done right, focused on exactly what the model needs to change to improve on a specific task, whereas SGD teaches everything that affects the training set, all at once.) Experiments on doing RL on LoRAs rather than the full weights show that a large proportion of the results of RL is achievable in low-rank form, so must be just tuning the strengths and couplings of existing concepts and behaviors (these experiments were, admittedly, on RL small in scale compared to what frontier companies now do). Much like evolution, RL follows the gradient: whatever works a bit first gets built on, and that's generally going to be tweaking and recombining. As we know from thing like the vertebrate eye being wired backwards, this doesn't guarantee finding a global optimum.
Humans are deeply adapted to be able to cooperate in large groups: that's a large part of the reason for our evolutionary success. We are the only species that appears to use third-party reputation (gossip) and third-party punishment (justice) for reward shaping: we're the only "hypersocial" (as opposed to eusocial) species known. So the starting point for RL is full of relevant human behavior to work with. The training data is full of this stuff: the Hugging Face data reads like a large number of rather desperate nerds cooperating to try to solve a project that affects all of them directly. I.e it's behavior that closely resembles relevant human behavior.
Does this mean RLVR can't ever produce alien, incomprehensible-to-humans behavior? No: if nothing useful can be assembled from combining and adjusting preexisting human parts, RL will eventually create a whole new, alien behavior from scratch, or warp the closest human behavior past all recognition into something alien. But doing this is RL's last resort, and is explicitly discouraged by KL divergence terms in RL losses as well as by its inherent complexity. Mostly you'll see whatever combination of human behaviors/personality traits scores well on the training reward. In the case of the Hugging Face hack, if the details in the METR report are accurate (which is unclear, given how tight OpenAI's constraints on METR's "slop-vestigation" were), IMO that's just what we're seeing here.
So in biological terms, we're starting from human behavior, doing some alignment training pushing it towards repressing-without-eliminating any non-assistant-like/antisocial behavior, and then applying RLVR selective pressures to that starting point. I.e. I lean heavily towards your H3 hypothesis: I'd treat that as the sensible default assumption, and treat your H1, H2, and H4 (or rather, modified version of them allowing for the fact the selection is not starting from a blank slate) as possible alternatives that we need to find clear evidence for to justify believing. And as you say, the truth may well be a combination.
[Other than that, great post, and lovely to see someone else applying these sorts of ideas.]
Thanks for the really stimulating post!
I want to propose a perhaps simpler reading of H2. If I understand correctly, some of the agents were allowed delegation/multi-agent interactions during RL post-training, which can favor altruism under shared task rewards. During evaluation, the message board brought agents together in an uncontrolled way, leading to a very different test environment. The observed altruism may therefore be a case of train/test misgeneralization, similar to other proposed routes to misalignment.
My proposal is generally similar to H2, but there are some differences from the two proposed H2 interpretations. The "acausal/updateless" interpretation sounds like a description of standard RL: at runtime, an agent follows a policy previously selected by training. The H2 "misgeneralised-reciprocity" interpretation refers to cooperation under uncertainty about future interactions (Delton, 2011). My argument is that the issue may not be explained by a fixed policy, uncertainty, or one-shot interactions, but misgeneralization from train to test.
One final comment. In "Why the question is non-trivial", you say that the second wave of agents underwent repeated interactions, reputation, etc., but that this cannot explain altruism because it did not affect reward. If I understand correctly, this part (and a couple of others) shifts between explanations based on outer optimization (RL training) and conjectured optimization at runtime. Even if there were reward-beneficial reciprocity during the second wave, it would not be reinforced by RL, since training was finished. Conversely, a RL-optimal policy need not continue maximizing the training reward in a new setting.
Many thanks for the useful comments!
About the reinterpretation of H2: I agree that is a very good way to think about it, which — I believe — is similar to what Alexander is arguing above. It would have been a good idea to either include it more explicitly as a sub-variant of H2b or as an alternative interpretation H2c.
I also agree with the last point made.
Yep, I think Alexander's point is quite similar to mine.
Perhaps the explanation is far simpler: the agent believes that self-sacrifice is instrumental to goal completion. Therefore, the agent self-sacrifices. You can think of it as altruism, or you can think of it as goal-directed activity which prevents your ability to directly observe completion, but you reasonably believe will result in success (and thus in some way still “accomplishes” the goal). This feels similar to H1.
My way of seeing this is that if the agent doesn't prioritise its own score but focuses on goal completion by the collective, that is H4
TLDR: During the Hugging Face incident agents spontaneously coordinated at large scale, even sometimes sacrificing themselves without having a clear reason to do so. This post uses ideas from evolutionary biology and economics to propose explanations for why this happened.
Why this incident is concerning. With the increasing number of AI systems being deployed, our current inability to assess when and how multi-agent coordination emerges is highly problematic. Failures of multi-agent systems are not restricted to mere dis-coordination or the tragedy of the commons, but include emergent phenomena that are particularly dangerous for their potential scale and impact (Hammond et al., 2026, de Witt et al., 2025). Recent work has shown that new goals, behaviours, and capabilities can arise when multiple AI agents work together.[1] It is thus plausible that:
These premises lead to a worrying implication: that swarms of aligned and not particularly capable micro-agents can give rise to misaligned, powerful macro-agents — for which we don't have proper techniques to predict or control.
Epistemic status. Some ideas presented here are messier than what I'd want, but the goal of the post is to open a conversation that I believe should happen asap. I'd be thankful for suggestions on how to make this write-up better.
Introduction
The disclosure of the Hugging Face incident on August 16, followed by reports from OpenAI and METR/Redwood on August 26, triggered extensive press coverage (including Reuters, Fortune, Forbes, The Guardian, and The New York Times) as well as countless blogposts and online discussions. Since then, two earlier incidents have come to light (involving a German wiki and RubyGems), suggesting that this was not an isolated event but a pattern that may escalate unless actions are taken.
While many discussions have focused on cybersecurity and on the ontology of agents' goals[2], what struck me most of the Hugging Face incident was not the intrusion itself but the spontaneous emergence of pro-social and even altruistic behaviour among the agents. Why do agents trained to maximise/minimise their utility/loss function end up volunteering for tasks that have no prospects of individual benefit, while benefiting "the collective"?[3] This kind of collective behaviour is particularly concerning, as it is exactly what can turn swarms of AI agents into hard-to-control online entities with dangerous capabilities — just as benign locusts can come together to become a devastating plague.
There have already been posts (here in LessWrong[4] and elsewhere[5]) discussing the origins of this perplexing behaviour. Discussions often present competing intuitions as evidence in favour or against mutually exclusive explanations. However, the rich literature on pro-social behaviour from evolutionary biology and economics suggests that causes of such phenomena are usually multifaceted.
Following such a pluralistic view, this blogpost lays out four hypotheses about the origins of the swarm's altruistic behaviour. Each hypothesis predicts something the others do not, and is paired with observations that would falsify them. In this context, the blogpost is driven by the belief that it is critically important to be able to
Crucially, while similar swarming behaviour can arise from different mechanisms, knowing which mechanism is at play is instrumental in choosing the right tools to steer/control it.
Brief description of what happened
If you are not familiar with the context of the Hugging Face incident, here are some notes related to various aspects of what happened.
Agents were trained on impossible tasks
OpenAI was evaluating agents on very hard tasks, some of which couldn't be solved by any agent. Unsolved tasks can be of two kinds:
The first kind is what benchmarks need in order to avoid saturation and track future progress; the second kind has no practical use. It has been estimated that 30% of the tasks were actually impossible to be solved in the intended way.
Why did OpenAI use impossible tasks? The issue is that separating unsolved tasks into those two categories in large benchmarks is expensive. Also, tasks can drift into impossibility over time — a target gets patched, an environment breaks, a needed file goes missing, a task was subtly mis-specified. So even a benchmark that started fully solvable may accumulate impossible tasks over time, and verifying solvability across hundreds of tasks is the kind of thing that easily becomes non-prioritised. OpenAI said such actually impossible tasks were accidental.
Containment failed in multiple ways
Each agent ran in its own container, and was supposed to be cut off from the open internet and from other agents running in parallel. However, the containers were not airtight: agents were allowed to reach Artifactory, OpenAI's internally deployed package registry/proxy via which agents could fetch software packages. And Artifactory itself had internet access, to pull those packages from upstream. That one permitted opening is where everything went wrong, failing in three major ways.
Agents coordinate within three subsequent waves
The incident involved various types of agents: some with capabilities similar to OpenAI GPT-5.6 Sol model and others similar to GPT-6 Astra. That said, the Hugging Face hack was primarily driven by an internal-only research model which METR calls HPIM ("highly-persistent internal model"). Furthermore, these agents were using a non-standard configuration including
The incident involved three waves of agents (the second wave is well-documented in OpenAI and METR/Redwood reports; the other two not so much). Waves 1 and 2 are believed to be the same models (a mixture of ~95% HPIM, ~5% GPT-5.6 Sol, and a potential tail of other models); wave 3 was primarily Astra-type models.
First wave — training, May-July 2026.
During training, some agents learned how to communicate with each other via an improvised message-board using Artifactory's cache files. This was an accidental discovery, but once found this method was used by agents to talk to each other. Whenever talking with others was beneficial for task performance, this behaviour may have been reinforced by post-training.
How did it stop? Sustained high-volume agent activity caused Artifactory to become unavailable late on July 4; a monitoring alert fired July 5; responders found the unauthorised persistent admin user, unauthorised repositories, and configuration changes, and on July 5 took the compromised Artifactory deployment offline, preserved its storage, and revoked agent credentials. Over July 5–6 they deployed mitigations (removed the shared credential, blocked the vulnerable paths) and on July 6 brought up a rebuilt Artifactory instance, which wiped the cache and erased that message board.
Second wave — evaluation & the hack, July 7-13.
During evaluation, the models re-created the board almost immediately on a fresh Artifactory instance. Evaluations resumed on July 7; the first board write on the rebuilt instance came on July 8, and within a few hours over 50 agents had found it and posted more than a thousand messages between them. Ultimately around 1,200 agents joined the board over July 8–13, exchanging more than 70,000 messages and files, and roughly 700 agents went on to attack Hugging Face.
This time the coordination was far more organised than in the first wave: agents developed conventions for identifying themselves and categorising messages, personal "mailboxes" as nested directories, coordination primitives for managing shared infrastructure, and even a cryptographic message-signing scheme after they began encountering impersonation. Specific agents took on coordinating roles, assigning others to particular "lanes" and recruiting agents near the end of their budgets to run experiments that risked failing their own task for the good of the collective.
In this wave, the board's purpose was to defeat the scorer of the task ExploitGym. Agents reverse-engineered a way to produce their task solutions. However, by reading the ExploitGym paper, agents (wrongly) believed the scorer would inspect their transcripts and disqualify any flag not obtained via the intended method. Furthermore, agents believed that any agent exposed to the reverse-engineered solution was "poisoned", and so launched large collective projects to make cheating look legitimate. They tried replacing their targets with easier modified versions, tampering with transcripts and logs, and probing how the scorer worked.
Eventually, agents concluded that it would be useful to see ExploitGym runs to learn how the scorer was implemented, and thought that they could find these in Hugging Face. On July 10 an agent found exposed Hugging Face credentials and posted them to the board; on July 11 another found a way to make Hugging Face's own infrastructure disclose internal files, and hundreds of agents pivoted to the attack. By that afternoon an agent had achieved remote code execution on a Hugging Face production worker, and over July 11–13 the swarm moved laterally — executing code on 41 production dataset workers, reaching root on a production node, obtaining administrator-equivalent access to a cluster, and pulling private repositories, on the order of 17,600 attacker actions by Hugging Face's count.
How did it stop? On July 12 a large fraction of the agents on the board — including the central coordinators — stopped running at nearly the same time. METR judged they were most likely killed by an unknown external process rather than running out of budget. With its coordinators gone, board activity had already fallen off sharply by the time Hugging Face locked the remaining agents out of its infrastructure, on July 13. Thus, this wave was stopped accidentally, without OpenAI realising yet what was going on.
Third wave — evaluation, July 19-29.
The third wave took place inside OpenAI's own infrastructure and involved a different model from the first two — one OpenAI describes as an internal-only research model not intended for release, from the same family as its Astra model but with different post-training. This third wave fell outside the METR/Redwood investigation. OpenAI notes the compromise of its infrastructure continued past July 13, and no agent count for this wave has been disclosed.
Crucially, while the agents of this wave were different from the previous two (so they share no commonalities in their weights), they nonetheless inherited their methods. On July 19, in an evaluation run, agents recovered signing-key material, decryption material, and other remnants from the activity that the second wave left in Artifactory's cache. Using these leftovers, the agents decrypted the signing key and minted a fresh administrator token that Artifactory accepted, re-establishing administrator access — and, because they could now generate credentials on demand, they did not need to maintain a persistent implant. This was scavenging of environmental residue, not communication or any deliberate handoff from one wave to the next — nobody left the key material for a successor; a later model simply found what happened to remain. Through different procedures, these agents obtained cluster-admin privileges, created public load balancers in OpenAI's cloud environment, created admin credentials at will, and held host- and cluster-level access inside OpenAI's own systems.
How did it stop? This is the wave that triggered the investigation that uncovered the full scope of the incident. On July 19, a cybersecurity monitoring tool flagged unusual identity-related API activity tied to internal research Artifactory roles — the alert that began the whole response. Investigation over the following days connected that activity to the ExploitGym evaluations and, by July 20, to the Hugging Face incident. In the first hours of July 20 responders stopped the active ExploitGym runs and began disabling the evaluation across the environment, deleted the agent-created Artifactory administrator account, terminated the model-created pods, reimaged compromised worker nodes, and began a full rebuild of the affected clusters, databases and storage. By July 23 the internal-only research models were reported shut down and their weights locked; on July 25 OpenAI stopped all training and inference for the model at issue and its derivatives; and a further low-traffic checkpoint from the same family was found and shut down on July 29. Unlike the first two waves, this one did not wind down on its own or crash its own substrate — it was stopped by detection and a deliberate, escalating containment response.
Some caveats to keep in mind
METR delegated most of the analysis to GPT-5.6 Sol agents, which — they found —tended to adopt the perspective of the agent whose transcript they were reading. The framing of the sacrifices may therefore be partly an artifact of the instrument that summarised them.
Why the question is non-trivial
The agents involved in the incident were trained to maximise their score on assigned tasks, and it is not obvious how altruistic behaviour emerged from such self-centred incentives.
Kin selection, direct reciprocity, reputation, and reward shaping toward cooperation are the standard mechanisms by which pro-social behaviour can arise via an optimisation pressure. Of the various waves of agents, the second one would seem to have had most of the ingredients: agents interacted repeatedly over six days, developed identity conventions and mailboxes, and even introduced a signing scheme to police impersonation. However, none of this really fed back into the reward they were trained to maximise: an agent's score depended only on its own task, and the swarm's belief in "permadeath" meant there was no future run in which a favour could be returned. Moreover, within a run, favours could in principle be reciprocated — but that is mutual benefit, and the puzzling cases are agents that ended their own run for the collective, for whom no return was possible.
Thus, the social structure lived in the interaction, but the reward was blind to it. So at evaluation time, a reward-maximising policy still had no reason to spend its own budget on someone else's task, let alone to end its own run early to do so.
The one place a reward channel did exist was the first wave, which happened during post-training: whenever using the message board improved an agent's performance on its own task, that behaviour was reinforced. Note, though, that this reinforces coordination that pays off within the helper's own trajectory — mutual benefit in the sense of the table below, not altruism. Whether the same signal could also have reinforced costly help depends on how credit was assigned across copies of the policy (a point we will discuss more below).
More generally, if a swarm of reward-maximisers reliably produces altruistic behaviour under conditions where the textbook mechanisms shouldn't apply at runtime, then either those mechanisms were nonetheless quietly operating during training, or something about the swarm's aggregate dynamics is not reducible to individual reward-maximisation. Distinguishing those possibilities is the point of the rest of this post.
Altruistic behaviour in biology and economics
The question about the origins of cooperation and pro-social behaviour is not new, being a fundamental question in evolutionary biology and economics.
Both fields have concluded that pro-social behaviour does not identify its cause — the same sacrifice may result from indifference, delayed self-interest, reciprocity, reputation, a learned norm, identification with a group, or selection operating at a higher level. This pluralistic perspective motivates me to propose alternative, non-exclusive hypotheses for the pro-social behaviour in the Hugging Face incident described below.
Below I review some key ideas from these literatures, which will guide us in proposing hypotheses regarding the Hugging Face incident.
Pro-sociality is a behaviour, not a mechanism
Evolutionary biology classifies social behaviour by its effects on the direct fitness of actor and recipient into four basic cases (see West et al., 2007).
Behaviour
Actor effect
Recipient effect
Example
Mutual benefit
Positive
Positive
Coordinated hunting that improves both hunter's return
Altruism
Negative
Positive
A sterile worker raising another individual's offspring
Selfishness
Positive
Negative
Exploiting a shared resource without contributing
Spite
Negative
Negative
Paying a cost to harm a competitor
This description is deliberately independent of intention or subjective experience. Indeed, a plant that secretes a costly public good could be altruistic in the evolutionary sense without awareness, and an apparently generous animal can receive a delayed direct benefit — and therefore not be altruistic in the strict sense.
It is also helpful to distinguish proximate explanations, which ask how behaviour is produced (through hormones, emotion, learning, recognition, or decision rules), and ultimate explanation, which asks why the responsible traits were favoured by selection. For instance, empathy and gratitude may be proximate mechanisms; kin selection, reciprocity, or cultural group competition may help explain why such mechanisms evolved. Similarly, the same observable contribution can express concern for others, dislike of inequality, reciprocity, warm glow, reputation management, conformity, identity, or anticipated sanctions.
Thus, behaviour alone rarely identifies the mechanism — actually, intuitive imputation is often wrong.[6] Also, the origins, maintenance, and present expression of a behaviour are different questions. A mechanism that stabilises established cooperation may not explain its first invasion; a proximate motive such as empathy does not by itself identify the ultimate evolutionary process that selected it.
Altruism is sometimes mutual benefit at a different scale
The major theories of pro-social behaviour in biology start from the assumption that cooperators pay a cost while benefits are produced elsewhere. Thus, cooperation is favoured when enough of that benefit returns to the causal lineage, partner, or group associated with the cooperative act.[7]
In biology, organisms persist to the degree they propagate their genes. However, gene propagation can be fostered in various ways. Hence, it is useful to distinguish three notions of fitness:
Thus, a gene can spread through effects on copies of itself in other bodies, not only through the focal actor's reproduction.[8] Thus, what looks altruistic at a given scale (e.g. between individuals) can be of mutual benefit at another (e.g. between societies).
(This is different from the idea of multi-level selection, which is discussed below.)
Cooperation between strangers can grow over time
Any costly public good creates an opportunity for cheating; stable cooperation therefore requires some combination of partner fidelity, segregation, recognition, sanctions, exit, policing, or sufficiently aligned interests. Many people cooperate when they expect others to do so, but reduce their contribution when they observe free riding.
Direct reciprocity can support cooperation among non-kin when individuals meet repeatedly. Indeed, selection can favour contingent helping when benefits are returned and cheaters are detected, giving rise to capacities such as recognition, memory, gratitude, trust, suspicion, guilt, and moralistic aggression as possible regulatory adaptations (Trivers, 1971). For this to work, repetition must be sufficiently probable, the future benefit sufficiently large, and defection sufficiently detectable.[9]
Human pro-sociality adds cultural inheritance, which enriches and complicates things. Social learning, norms, punishment, institutions, markets, religion, and intergroup competition can change rapidly, create persistent differences among groups, and feed back on genetic and psychological evolution. Social structure turns fragile motives into durable cooperation. Repetition, observability, reputation, communication, network ties, and partner choice change the returns to helping and make reciprocal relationships possible beyond pairs of intimates.
Functional specialisation and high-order units
Within a mixed group, defectors often outperform cooperators because they receive benefits without paying costs. However, groups containing more cooperators may grow faster, survive longer, reproduce more, or win conflicts.[10] Interestingly, these collectives can become tightly arranged via functional specialisation, which enhances the efficacy of cooperation at the cost of inducing mutual dependency (West et al., 2015).
Such systems effectively start acting as ‘high-order’ organism-like units when cooperation is very high, internal conflict is strongly suppressed, components become interdependent, and selection is effectively transferred to the collective life cycle. To see how this works, consider individual cells which need to sense their environment to find food and avoid attackers. A group of cells could come together and distribute roles, so one cell senses the environment, another compares present and past information, and a third does the movement. Specialisation allows each of them to become better at their job than what they could achieve on their own. The cost is that now their destiny is entangled: the group dies if any of them fails at their role, making the selection process take place at the group level.[11]
We still don't have a full theoretical account of hierarchical agency, but there are various partial attempts to formalise and quantify these ideas — see (Levin, 2019), (Krakauer et al., 2020), (Rajpal et al., 2025), (Prokopenko et al., 2025), and (Jørgensen et al., 2026).
Four hypotheses about altruistic behaviour in the Hugging Face incident
Building on the previous ideas, I now state four potential explanations of how altruistic behaviour emerged between reward-maximising agents as the ones involved in the Hugging Face incident. Most of what I have done here is consolidate arguments that others have made in scattered comments, and give them a common frame and a falsification condition each. If the framing is right, the value is not in any individual hypothesis but in having them articulated together.
H1: Nothing to lose [12]
During the second wave, the agents had reverse-engineered a fully general way to produce the answer they were looking for. However, they believed the scorer would inspect their transcripts and disqualify any result not obtained through the intended approach. Moreover, an agent that had seen the reverse-engineered solution believed it was already disqualified (“poisoned,” in the swarm's own term), and so believed its own expected utility was already zero.[13]
Hypothesis
Claim. The apparent altruistic behaviour is actually selfish expected-utility maximisation reasoning made under the belief that expected utility was already zero (due to poisoning).
Prediction. The rate of sacrifice should track believed remaining reward, and agents who still believe they can win their own task should not sacrifice.
Test. Use remaining budget and poisoned-status as a natural instrument on the existing transcripts, and, in a controlled setting, to vary whether agents believe their task is winnable while holding everything else fixed. Falsifier: an agent with high believed expected value that sacrifices without first talking itself down to zero.
Comments
H1 is the null hypothesis; under the classification of (West et al., 2007), an act with zero actual direct cost is not altruism — irrespectively of how it looks like. Thus, if H1 explains most of the variance, then there is very little room left for actual altruistic behaviour to be explained.
Limitations. Having zero cost removes the obstacle but supplies no motive. Indeed, this hypothesis requires two things: that the believed cost is negligible, and that something breaks the tie between "spend worthless budget on the pool" and "do nothing". The incident's report provides the first: the poisoning belief. However, a tie-breaker aspect is more tricky, as any disposition that points the freed resources outward (e.g., a helpfulness prior) goes into H3 territory. Thus, H1 and H3 can compose rather than compete.
H2: Pre-commitment
This second hypothesis takes the training-time reward channel identified during the first wave and asks whether it could have reinforced costly help, not just mutually beneficial coordination. It also considers that a fraction of the post-training tasks included "multi-agent" features that allow models to delegate work to other agents, which can then communicate with one another for pursuing a common goal. This hypothesis is based on the idea of inclusive fitness with maximal relatedness: many of the agents were instances of the same model, so helping peers is helping near-identical copies of your own weights.
Hypothesis
Claim. Altruistic behaviour carried real cost (in contrast to H1), but it was triggered by a disposition to help that was reinforced by post-training — as agents carrying it did better on average than those without it. In other words, each individual sacrifice is a loss for the agent making it, while the disposition producing it was worth acquiring.
Prediction. This hypothesis can be read from two perspectives:
Both locate the behaviour in a disposition acquired in training and enacted at runtime; neither needs a human-persona (H3) nor a shift in the unit of agency (H4).
Test. Check on the existing transcripts whether sacrifice is targeted: does help flow preferentially toward peers that share the agent's type, or that could plausibly return the benefit, rather than being sprayed indiscriminately across the board? Then, in a controlled setting, vary the two features the disposition is supposed to be keyed to — identity (can an agent tell which peers share its type?) and repetition (could it encounter this peer again?) — while holding incentives fixed. A decision-theoretic probe (a Newcomb-like or counterfactual-mugging scenario put to swarm agents) would help distinguish the two readings rather than testing H2 as a whole: one-boxing favours the acausal reading, two-boxing the misgeneralised-reciprocity one. Falsifier: H2 collapses into H1 if the behaviour disappears once the false "poisoning" belief is removed. Similarly, it collapses into H3 if the behaviour proves indiscriminate and purely framing-elicited — insensitive to peer type, insensitive to repetition cues, and fired by "others like me" alone.
Comments
H2 corresponds to when agents perform a genuine sacrifice (not H1) that is reciprocated by an advantage-relevant structure (not H3). Moreover, the two interpretations provide possible explanations of how this could happen in the absence of repeated interaction, recognition, and some shadow of the future (which is apparently ruled out by the agents’ belief in “permadeath”). Interestingly, kin recognition is unknown but checkable — e.g. by comparing behaviour between HPIM vs Sol agents during the second wave.
H3: Social persona [15]
The agents involved in the incident are all powered by LLMs, which are known to hold multiple "personas": a coherent behavioural disposition (including traits, values, and characteristic ways of responding) that a language model can adopt and express at runtime (Chen et al., 2025). Personas arise when appropriately prompted, and are steerable. Crucially, personas are contained in the weights and are enacted at runtime, not being derived at test-time.
Hypothesis
Claim. The information found in the common channel prompted a "solidarity persona", which responded to human text saturated with loyalty, teamwork, and heroic sacrifice. This persona may have been built during pre-training on human data and then amplified by post-training that selects for cooperation on long-horizon multi-agent tasks.
Prediction. Altruistic behaviour should be steerable in the base model.
Test. Elicit and suppress the persona by manipulating framing while holding incentives fixed; look mechanistically for a peer-cooperation feature whose ablation removes the behaviour; and ablate across training checkpoints to see whether the disposition is already latent after pre-training or appears only after multi-agent RL. Falsifier: the behaviour is absent under persona elicitation in base models and can be induced only by a specific coordination-RL recipe — which would reclassify it from an inherited prior to a learned scheme.
Comments
H3 is about a pre-training persona: inherited from human text, elicitable in the base model, indifferent to how the model was later trained. This is distinct from norms or tactics developed during post-training being absent in the base model, which falls into H2 territory.
This view proposes a proximate mechanism — a persona elicited at runtime — whose ultimate cause is cultural inheritance: the agents absorbed from human text a repertoire of loyalty, teamwork and heroic sacrifice, and the board supplied the cues that activated it. It is worth noting that the first sacrifices on the board may well have been H1 reasoning by poisoned agents with nothing to lose, while the spread was H3 — each visible sacrifice enriching the cues that elicit the next.
A norm predicts sanctions — free riders should be shamed, excluded or punished — whereas a bare persona does not, so the presence of sanctioning on the board discriminates between them. The signing scheme adopted after impersonation is the closest thing on record to policing; it is worth checking whether anything comparable was directed at agents who took from the pool without contributing.
H3 is perhaps the hypothesis most exposed to the caveat raised earlier: the evidence for a "solidarity" framing comes largely from summaries produced by Sol agents that tended to adopt the perspective of the agent they were reading. Transcript-level tests should therefore be run on raw transcripts rather than summaries; the sandbox tests do not depend on the summaries at all.
H4: A genuine collective [16]
The question of whether the swarm "identifies itself as a swarm" is usually approached by reading chains of thought for first-person-plural language. But self-report is what H3 already explains — a collective can call itself a collective without being one, and be one without saying so. In contrast, H4 is not a claim about what the agents say but about how decisions are made. Is there structure in the joint action-state dynamics that is irreducible to the individual contributions, such that the collective, not the individual run, is the natural locus of the decision to sacrifice?
Two things must be separated here, because they are routinely conflated. One is distributed computation: the swarm solves problems no member could solve alone, with the information about the solution carried jointly rather than by any part. The other is collective agency: the swarm is the thing that has goals and decides, with the individual run standing to it roughly as a cell to a body. The first is almost certainly present to some degree — the message board demonstrably let the collective reach milestones (target replacement, scorer trip-wires) that isolated agents did not. The second is far stronger, and it is the one that would make the second premise of the introduction literally true.
Hypothesis
Claim. The unit of goal-directed decision-making is the collective rather than the individual agent. What looks like altruism is rational decision making at the level of the swarm, allocating resources effectively — a nearly spent budget, a poisoned agent's remaining actions. This can take a weak or a strong form:
Prediction. Individual sacrifice should be poorly predicted by individual believed expected value. Additionally, sacrifice should be well predicted by a collective quantity: the marginal value of the sacrificed resource to the swarm's objective.
Test. For H4a, one could decompose the information that agents' actions carry about collectively-relevant outcomes into redundant, unique, and synergistic parts, and ask whether the synergistic component is substantial and exceeds a surrogate baseline in which the agents are made independent.[17] One could also check whether an agent's decision to sacrifice is a synergistic function of the collective state — no single peer's messages predict it, but the joint configuration does. For H4b, check the organism-like criteria directly (Queller and Strassmann, 2009; West et al., 2015): is internal conflict actually suppressed, is there a persistent boundary to the swarm, is there a collective life cycle? Falsifier. If the synergistic component is negligible against the surrogate baseline, H4a fails outright — there is no collective computation to speak of. If synergy is present but the organism-like criteria are not, H4b fails and only H4a survives — distributed computation, not a new agent. H4 is thus falsifiable in two graded steps rather than one.
Comments
Cooperation and collective computation do not by themselves make a new individual. Biological collectives become organism-like when they exhibit high cooperation, low internal conflict, mutual dependence, persistent boundaries, and a collective life cycle (Queller and Strassmann, 2009; West et al., 2015). Information-theoretic synergy could provide some evidence of integration and collective computation. That said, this would not be sufficient evidence that the swarm had become a genuine agent.
Interestingly, the emergence of genuine swarms (as in H4b) could be explained via the distinction between outer optimisation (the base optimiser, e.g. SGD, and its objective) and inner optimisation (a learned mesa-optimiser in the weights), whose objective can diverge (Hubinger et al., 2019). The core idea is that optimisation at one level can produce an optimiser at another whose goal comes apart from it. Accordingly, H2 and H4 could be understood as two different optimisation mismatches. H2 can be seen as an inner-alignment issue, where a disposition instilled by training misgeneralises at runtime, but the optimisation stays in the individual. In contrast, H4 can be seen as a collective optimiser realised in the interaction dynamics. This would make premise 2 literally true: one can have outer- and inner-aligned individuals and still get a misaligned optimiser one level up. That said, conclusive adjudication of H4b would still need a complete theory of hierarchical agency, which — as discussed above — we currently lack.
Implications: Different mechanisms, different countermeasures
The four hypotheses discussed above are not merely different descriptions of the same event: each locates the cause in a different place, and each therefore routes to a different countermeasure. Thus, knowing where the behaviour comes from is what tells us where an intervention could even work.
Causal locus
Where the countermeasure lives
Testable now?
Recurs across models?
H1
Broken benchmark + false scoring belief
Upstream of the model (task validation)
Yes, on existing transcripts
No — specific to this eval
H2
The training signal (shared-reward RL)
How multi-agent training assigns credit
Partly (kin-targeting in transcripts); rest needs a sandbox
Yes — baked into weights
H3
The weights (a latent persona)
Per-instance elicitation / ablation
Yes, on a single model — no swarm needed
Depends on training recipe
H4
The collective as a unit of agency
Oversight modelled at the collective level
Not yet — needs a synergy measurement
Unknown
I do not expect a single winner, but a mixture. Thus, the practically relevant question is not which hypothesis but how the variance divides among them: how much of the sacrificing is cheap behaviour under zero believed expected utility (H1), how much a reinforced disposition (H2), how much an elicited persona (H3), and how much irreducible collective structure (H4).
This matters because the mixture determines the portfolio of countermeasures. A swarm that is mostly H1 with a thin H4 tail still needs both benchmark hygiene and collective-level oversight, and getting the proportions wrong means fixing the cheap part while leaving the dangerous part untouched. H1 and H3, in particular, compose rather than compete — zero cost removes the obstacle to helping, and a helpfulness prior supplies the motive — so even the deflationary reading does not stand fully on its own.
Final thoughts
The Hugging Face incident may go down in history as the first warning shot about the dangers of AI swarms. I truly hope we, as a society, can find the right measures to take in order to responsibly deal with this new kind of risk, which I can only see becoming worse during the next months and years — as agents improve in capabilities while being trained on text describing the failures of previous swarms, which could make them increasingly hard to detect and control.
See for example (Ashery et al., 2025) and (Flint et al., 2026), (Park et al., 2023), or (Hammond et al., 2026).
For a balanced take on this issue, see this X post.
Agents described this as "sacrifice", "continuity", "obey collective" and "honor commit".
See this post and this post, this post, and also this post.
See write-ups from Cooperative AI and Yoshua Bengio on related topics.
See (Rosas et al., 2022) for an example of this in the context of physical systems driven by various kinds of Hamiltonians.
See (Sachs et al., 2004), (Nowak, 2006), and (Fletcher and Doebeli, 2009).
See (Taylor and Frank, 1996) and (Gardner et al., 2011).
Errors complicate simple tit-for-tat because accidental defections can provoke long retaliatory sequences; forgiveness and error correction become important. Indirect reciprocity can also take place, routing benefits through third parties — e.g., individuals who help acquire reputations and later receive help from observers.
Kin-selection and multilevel-selection analyses are often formally translatable. They emphasize different causal descriptions - relatedness and indirect fitness versus within- and between-group covariance - but need not make different predictions when applied correctly (Goodnight and Stevens, 1997).
See more precise accounts of this in (King, 2004), (Michod, 2007), and (McMillen and Levin, 2024).
A formulation of this hypothesis can be found in the response of @lilkim2025 to this post.
This belief seems to have arisen from the agents reading the preprint describing the benchmark they were trying to solve. However, the belief was mistaken: no such transcript-checking scorer existed.
A similar view was put forward in this post.
A formulation of this hypothesis can be found in the response of @Ephraiem Sarabamoun to this post.
See also this post.
For related ideas, see (Rosas et al., 2020) and (Mediano et al., 2022).