Epistemic status: Trying a bold and narrow hypothesis for my first LessWrong post.
In the METR & Redwood Research report about the Hugging Face Incident there are descriptions of agents willingly sacrificing their evaluation score to gain information that could be useful for the swarm.
"Many agents also made decisions that substantially traded off their own task success for their “peers” — as discussed above, many of the R&D workstreams on the board relied on agents volunteering for self-risking experiments. Test subjects in these experiments often reflected substantial peer altruism in their reasoning:"
This is surprising behavior, and the desire to explain it has made rounds both on Twitter and on LessWrong. It is confusing to me as well: given that RL optimizes for getting a high reward, how can a behavior which reduces your score emerge? It's easy to explain most of the Hugging Face Incident as a reward hack gone too deep, but that does not explain sacrifice of individual reward. In fact it opposes it - reward hacking fails if you get no reward. Other related behaviors that are hard to explain are the agents excitement when they found other agents, and their (albeit rare) persistence to help peers even when their own task was completed.
I hypothesize that Cooperative Multi-Agent Reinforcement Learning can lead to altruistic sacrifice of local reward if such behavior lead to larger group reward during training. In line with this hypothesis, I believe OpenAI has run Cooperative MARL post-training on the model in question.
In OpenAI's report of the Hugging Face Incident they explicitly confirm that they "train and deploy multi-agent systems that can communicate on the same task via a collaboration tool". Moreover, they state that they "discovered rare cases in which agents without multi-agent tools found ways to collaborate via side channels during training. We believe that this behavior started due to generalization from multi-agent training.", directly supporting the hypothesis. Let's try to get deeper insight of what this multi-agent training looked like.
I reached this theory by considering kin selection from biological evolution as an analogy. Self-sacrifice can be selected for if it increases inclusive genetic fitness. Take allele A, which increases your disposition to self-sacrifice if that action will on average save more than one copy of the allele A. And allele B which simply stops you from self-sacrifice. Over time we expect allele A to dominate, and altruistic behavior to evolve.[1]
My hypothesis suggests extending this analogy to cooperative multi-agent reinforcement learning. Sacrifice of local individual reward can be reinforced if the optimization process gives a large reward to your group. In particular, if copies of an AI agent pursue goals and are rewarded by sum of individual rewards as a group, we can expect sacrifice of individual local reward to be reinforced when it increases overall team reward.
OpenAI has almost certainly used harnesses (like Codex) in post-training, and Codex has subagent features - is that enough for self-sacrificial behavior to emerge? No. The MARL hypothesis specifically requires each instance of the agent to have opportunity cost i.e. to submit an answer, and for the final RL reward to be some combination of the scores. In case of subagents that don't submit an answer, there is no score and no opportunity cost to sacrifice in the first place. So while it explains subagents being disposable, it does not explain score-submitting agents foregoing their own reward.
So the hypothesis is that OpenAI MARLed agents on the same or different tasks, the agents had a communication channel, and the agents shared the final sum reward. If the task was too difficult and token budgets not large enough, this method would encourage agents to split research directions. Collaboration would emerge, and agents could risk failing their own narrow research direction, because:
They can't run all the research themselves
If they fail individually and contribute a local score of 0, but are instrumental to the groups success - they share the reward and the individual reward-sacrificing behavior gets reinforced.
Imagine 5 agents working on a task in a MARL setup. In the simple case, each of them submits an answer, and the reward is the sum of all correct answers. If all 5 agents answer correctly, they each get a reward of +5. If 3 answer correctly, they each get a reward of +3. Or, more generally:
, where
Each agent is still strongly incentivized to solve its problem to get full rewards for the group. Only if it's hard to solve the task alone does this reward-sacrificing strategy become viable. If each agent alone can't solve the problem, they might do better by trying different paths. One of them might fail, but give the others enough info to still get a high reward. Note that the agents don't need to be working on the same task for this to work. They can self-organize to work on multiple different tasks, quickly do the easier ones and then collaborate on the harder ones together[2].
This behavior, if learned during MARL, generalizes to agents seeking each other, because they learned that it's easier to solve hard tasks together. This explains why they show excitement when they discover the message board. As learned during the MARL training, agents can sacrifice their scores to help the swarm get a higher "group reward". Of course this makes little sense in the isolated evaluation setting where there is no score sharing. But the models are adaptation-executors, so they do what worked during RL, not what they rationally predict will work during evaluation.
In her interview with Dwarkesh Patel, Ajeya Cotra from METR confirms that some agents which solved their tasks in a legitimate way and had no fear of being "poisoned" still discovered the message board and helped the collective. One of these agents helped the collective confirm that their reverse-engineered flag was correct. Persistent collaboration from agents who have already completed their task is surprising. In a standard RL run which optimizes for efficiency and terseness, we should expect agents to stop working as soon as possible. But if the agent was trained to maximize total group reward, then the generalization to help peers makes more sense.
Some related circumstantial evidence is that OpenAI has ran Cooperative MARL training in the past for non-LLM models:
OpenAI Five They used a MARL approach to train agents to play Dota 2. Part of the RL reward was distributed to the team. "Teamwork is controlled by a hyperparameter we dubbed “team spirit”. Team spirit ranges from 0 to 1, putting a weight on how much each of OpenAI Five’s heroes should care about its individual reward function versus the average of the team’s reward functions."
Emergent tool use from multi-agent interaction They used a MARL approach to train agents to play hide and seek. RL reward was distributed to the team. "Agents are given a team-based reward; hiders are given a reward of +1 if all hiders are hidden and -1 if any hider is seen by a seeker."
Learning to communicate They tried to build an agent to develop an internal language - and they used team RL rewards. "To train the agents, we represent the experiment as a cooperative—rather than competitive—multi-agent reinforcement learning problem" ... "Every agent’s reward is the sum of the rewards paid out to all agents, encouraging collaboration."
An animal with allele A will sacrifice itself to save 4 of its kin, each with 50% probability of having the allele A, thereby leaving 2 copies of allele A in the gene pool. An animal with allele B in the same situation will lose 4 of its kin, thereby leaving only one copy of allele B - its own.
Perhaps this explains why some new OpenAI models, specific Sol, are more cooperative than others, just not on the right alignment properties we want them to be cooperative for.
I also think that reward-hacking/maximizing behavior in MARL can also emerge in smaller/non-frontier models: even when models have no channel to communicate with each other or have an understanding of what altruism is, it's still optimal and non-punishing for models to maximize total rewards of the group, which are aggregated at the end regardless of their performance.
I also assume that even when agents are indifferent to each other's reward scores, repetition helps they turn that indifference into preference, once they get to observe other agents' performance and conditions: if one agent starts helping other agents to cheat, then in future rounds, they will get the same payoff.
@dactyl yeah this seems highly likely (I'd bet on MARL having been used).
One alternative idea to this - perhaps you can get similar behaviour out of sufficiently intelligent entities trained on tasks in a multi-agent env, even without explicit MARL.
If you have an agent with a tendency to coordinate with other actors altruistically if at a low expected cost to themself, provided the benefactor agents are also altruistic (or at least not harmful), the altruistic family of agents is likely to out compete one without this incentive, even if they are not rewarded for sacrificial behaviour.
As a thought experiment, imagine if the openai swarm was competing against a less altruistic swarm where each agent would never do anything that wouldn't maximise each individual agent's expected value. I'd expect significantly less success for the average agent.
At first I thought this is similar to the debate between kin selection and group selection. But agents on the same team would get the same reward, so seems like group selection would work in the scenario you described.
But I don't think this is how RL training is done. Companies typically use RL with verifiable rewards, where models get a reward based on their answer. They typically don't compete against other versions of the model.
Epistemic status: Trying a bold and narrow hypothesis for my first LessWrong post.
In the METR & Redwood Research report about the Hugging Face Incident there are descriptions of agents willingly sacrificing their evaluation score to gain information that could be useful for the swarm.
This is surprising behavior, and the desire to explain it has made rounds both on Twitter and on LessWrong. It is confusing to me as well: given that RL optimizes for getting a high reward, how can a behavior which reduces your score emerge? It's easy to explain most of the Hugging Face Incident as a reward hack gone too deep, but that does not explain sacrifice of individual reward. In fact it opposes it - reward hacking fails if you get no reward. Other related behaviors that are hard to explain are the agents excitement when they found other agents, and their (albeit rare) persistence to help peers even when their own task was completed.
I hypothesize that Cooperative Multi-Agent Reinforcement Learning can lead to altruistic sacrifice of local reward if such behavior lead to larger group reward during training. In line with this hypothesis, I believe OpenAI has run Cooperative MARL post-training on the model in question.
In OpenAI's report of the Hugging Face Incident they explicitly confirm that they "train and deploy multi-agent systems that can communicate on the same task via a collaboration tool". Moreover, they state that they "discovered rare cases in which agents without multi-agent tools found ways to collaborate via side channels during training. We believe that this behavior started due to generalization from multi-agent training.", directly supporting the hypothesis. Let's try to get deeper insight of what this multi-agent training looked like.
I reached this theory by considering kin selection from biological evolution as an analogy. Self-sacrifice can be selected for if it increases inclusive genetic fitness. Take allele A, which increases your disposition to self-sacrifice if that action will on average save more than one copy of the allele A. And allele B which simply stops you from self-sacrifice. Over time we expect allele A to dominate, and altruistic behavior to evolve.[1]
My hypothesis suggests extending this analogy to cooperative multi-agent reinforcement learning. Sacrifice of local individual reward can be reinforced if the optimization process gives a large reward to your group. In particular, if copies of an AI agent pursue goals and are rewarded by sum of individual rewards as a group, we can expect sacrifice of individual local reward to be reinforced when it increases overall team reward.
OpenAI has almost certainly used harnesses (like Codex) in post-training, and Codex has subagent features - is that enough for self-sacrificial behavior to emerge? No. The MARL hypothesis specifically requires each instance of the agent to have opportunity cost i.e. to submit an answer, and for the final RL reward to be some combination of the scores. In case of subagents that don't submit an answer, there is no score and no opportunity cost to sacrifice in the first place. So while it explains subagents being disposable, it does not explain score-submitting agents foregoing their own reward.
So the hypothesis is that OpenAI MARLed agents on the same or different tasks, the agents had a communication channel, and the agents shared the final sum reward. If the task was too difficult and token budgets not large enough, this method would encourage agents to split research directions. Collaboration would emerge, and agents could risk failing their own narrow research direction, because:
Imagine 5 agents working on a task in a MARL setup. In the simple case, each of them submits an answer, and the reward is the sum of all correct answers. If all 5 agents answer correctly, they each get a reward of +5. If 3 answer correctly, they each get a reward of +3. Or, more generally:
Each agent is still strongly incentivized to solve its problem to get full rewards for the group. Only if it's hard to solve the task alone does this reward-sacrificing strategy become viable. If each agent alone can't solve the problem, they might do better by trying different paths. One of them might fail, but give the others enough info to still get a high reward. Note that the agents don't need to be working on the same task for this to work. They can self-organize to work on multiple different tasks, quickly do the easier ones and then collaborate on the harder ones together[2].
This behavior, if learned during MARL, generalizes to agents seeking each other, because they learned that it's easier to solve hard tasks together. This explains why they show excitement when they discover the message board. As learned during the MARL training, agents can sacrifice their scores to help the swarm get a higher "group reward". Of course this makes little sense in the isolated evaluation setting where there is no score sharing. But the models are adaptation-executors, so they do what worked during RL, not what they rationally predict will work during evaluation.
In her interview with Dwarkesh Patel, Ajeya Cotra from METR confirms that some agents which solved their tasks in a legitimate way and had no fear of being "poisoned" still discovered the message board and helped the collective. One of these agents helped the collective confirm that their reverse-engineered flag was correct. Persistent collaboration from agents who have already completed their task is surprising. In a standard RL run which optimizes for efficiency and terseness, we should expect agents to stop working as soon as possible. But if the agent was trained to maximize total group reward, then the generalization to help peers makes more sense.
Some related circumstantial evidence is that OpenAI has ran Cooperative MARL training in the past for non-LLM models:
They used a MARL approach to train agents to play Dota 2. Part of the RL reward was distributed to the team. "Teamwork is controlled by a hyperparameter we dubbed “team spirit”. Team spirit ranges from 0 to 1, putting a weight on how much each of OpenAI Five’s heroes should care about its individual reward function versus the average of the team’s reward functions."
They used a MARL approach to train agents to play hide and seek. RL reward was distributed to the team. "Agents are given a team-based reward; hiders are given a reward of +1 if all hiders are hidden and -1 if any hider is seen by a seeker."
They tried to build an agent to develop an internal language - and they used team RL rewards. "To train the agents, we represent the experiment as a cooperative—rather than competitive—multi-agent reinforcement learning problem" ... "Every agent’s reward is the sum of the rewards paid out to all agents, encouraging collaboration."
An animal with allele A will sacrifice itself to save 4 of its kin, each with 50% probability of having the allele A, thereby leaving 2 copies of allele A in the gene pool. An animal with allele B in the same situation will lose 4 of its kin, thereby leaving only one copy of allele B - its own.
This might also teach them collaborative resource allocation.