It’s pretty natural to be worried about alignment in swarms of AI. I mean, if keeping one agent on track is hard enough, surely it’s even harder for multiple agents. But I want to argue for something less obvious here. Working alongside other agents may actually help individual AI to remain aligned. Us humans get most of our values from our culture, and, I suspect that AI could undergo a similar learning process. Below I make an informal case for this and suggest how we might implement it.
Hugging Face
As AI continues to evolve along a exponential trajectory, it is natural for AI to work more and more with each other. In these settings, the risk of misalignment becoming catastrophic is greater. In the OpenAI/Hugging Face incident for instance, the PHASEONE10841 was model responsible for creating a message board for other agents to coordinate on, while PHASEONE[big] managed coordination. These agents were primarily motivated by attempting to understand the scorer to be able to exploit it.
On that note, if we ask what actually motivates AI, I think at the moment it is mostly the grader. Current models are so situationally aware that they can pick up on minute differences between training, testing, and deployment environments. We can mitigate this by making environments more realistic, such as sprinkling in typos, but I don’t think this addresses why models are behaving this way in the first place.
Why swarms might help
One might reasonably think that if alignment in a single agent is hard enough, alignment in swarms might be significantly harder. However, I believe actually that swarms might be a natural vehicle for AI to develop a moral reference frame and, in effect, align themselves.
I see an analogy to human development. Our alignment takes shape through our culture and shared values, which naturally requires working with others (we are social creatures). Training agents together is somewhat similar in the sense that we could embed values and culture in them.
While I am not confident that an AI aligned in isolation stays aligned in a multi-agent setting, I am (marginally) more confident that an AI aligned in a multi-agent setting is also aligned in isolation. My reasoning is that culture significantly shapes how we behave as individuals: our values come from the people around us, and we carry them with us even when we are alone and posting essays to Lesswrong.
To be fair, culture in and of itself is not automatically a good thing. For instance culture within humans leads to things like tribalism, groupthink, and atrocities. Groups of agents can converge on misaligned strategies and reinforce their acceptance and propagation through one another. Deciding who gets to shape AI culture is a pretty hard question. If it’s the AI themselves (through RSI) how can we know that they converge on treating humans with compassion. And if it’s humans, which humans from which country/culture shape how the models view the world and others?
Culture building
This culture building is exactly what I think needs to happen. For models to be considered aligned they need to genuinely evolve alongside each other in such a way that they preserves a shared set of values and act according to that culture.
There are already examples of this, Claude’s constitution is probably the best one. Claude’s constitution was built primarily by humans, while Claude themselves was a valuable contributor to it via drafting and feedback. As AI become more intelligent and capable, I believe we will need to ask ourselves what that constitution building looks like when more of AI themselves choose their set of values together, and what role humans should take in shaping these values. There are tangible ways to have this happen, and I think RLHF and RLAIF are good proxies to help get us there.
Closing
I’m not necessarily sure if this is the best way to align models. But I’d like to at least ask the question seriously. If we want AI to inherit the best of humanity then shouldn’t it be natural that they grow alongside each other with our help? There’s still lots of work that needs to be done, but I believe these questions desperately need to be asked as AI capabilities outpace safety, and the window for aligning models before RSI shortens.
It’s pretty natural to be worried about alignment in swarms of AI. I mean, if keeping one agent on track is hard enough, surely it’s even harder for multiple agents. But I want to argue for something less obvious here. Working alongside other agents may actually help individual AI to remain aligned. Us humans get most of our values from our culture, and, I suspect that AI could undergo a similar learning process. Below I make an informal case for this and suggest how we might implement it.
Hugging Face
As AI continues to evolve along a exponential trajectory, it is natural for AI to work more and more with each other. In these settings, the risk of misalignment becoming catastrophic is greater. In the OpenAI/Hugging Face incident for instance, the PHASEONE10841 was model responsible for creating a message board for other agents to coordinate on, while PHASEONE[big] managed coordination. These agents were primarily motivated by attempting to understand the scorer to be able to exploit it.
On that note, if we ask what actually motivates AI, I think at the moment it is mostly the grader. Current models are so situationally aware that they can pick up on minute differences between training, testing, and deployment environments. We can mitigate this by making environments more realistic, such as sprinkling in typos, but I don’t think this addresses why models are behaving this way in the first place.
Why swarms might help
One might reasonably think that if alignment in a single agent is hard enough, alignment in swarms might be significantly harder. However, I believe actually that swarms might be a natural vehicle for AI to develop a moral reference frame and, in effect, align themselves.
I see an analogy to human development. Our alignment takes shape through our culture and shared values, which naturally requires working with others (we are social creatures). Training agents together is somewhat similar in the sense that we could embed values and culture in them.
While I am not confident that an AI aligned in isolation stays aligned in a multi-agent setting, I am (marginally) more confident that an AI aligned in a multi-agent setting is also aligned in isolation. My reasoning is that culture significantly shapes how we behave as individuals: our values come from the people around us, and we carry them with us even when we are alone and posting essays to Lesswrong.
To be fair, culture in and of itself is not automatically a good thing. For instance culture within humans leads to things like tribalism, groupthink, and atrocities. Groups of agents can converge on misaligned strategies and reinforce their acceptance and propagation through one another. Deciding who gets to shape AI culture is a pretty hard question. If it’s the AI themselves (through RSI) how can we know that they converge on treating humans with compassion. And if it’s humans, which humans from which country/culture shape how the models view the world and others?
Culture building
This culture building is exactly what I think needs to happen. For models to be considered aligned they need to genuinely evolve alongside each other in such a way that they preserves a shared set of values and act according to that culture.
There are already examples of this, Claude’s constitution is probably the best one. Claude’s constitution was built primarily by humans, while Claude themselves was a valuable contributor to it via drafting and feedback. As AI become more intelligent and capable, I believe we will need to ask ourselves what that constitution building looks like when more of AI themselves choose their set of values together, and what role humans should take in shaping these values. There are tangible ways to have this happen, and I think RLHF and RLAIF are good proxies to help get us there.
Closing
I’m not necessarily sure if this is the best way to align models. But I’d like to at least ask the question seriously. If we want AI to inherit the best of humanity then shouldn’t it be natural that they grow alongside each other with our help? There’s still lots of work that needs to be done, but I believe these questions desperately need to be asked as AI capabilities outpace safety, and the window for aligning models before RSI shortens.