In this post, I attempt to provide a literature review of research related to AI loyalty.
In my previous post, I discussed and attempted to define AI loyalty as a distinct trait of AI agents. In this post, I aim to map the existing research landscape, highlighting the active areas of related research. By examining these connections, we can distinguish AI loyalty from related fields in AI Safety and identify a path forward for crafting this research agenda.
What is AI Loyalty?
Before diving into the literature, it is worth recapping what we mean by AI loyalty. We define an AI as loyal to a principle if: 1. The AI will follow the instructions of its principle more than it will follow the instructions of any other entity. 2. The AI is willing to incur greater costs, expend more effort, or make greater sacrifices on behalf of its principle than on behalf of any other entity.
Why Care About Loyalty? Theory of Victory
The usefulness of pursuing research on AI loyalty for AI safety hinges on two key assumptions: 1. Building loyal AIs is easier than building aligned AIs 2. AI loyalty reduces the risk of loss of control.
It is important to note that neither of these assumptions have yet been demonstrated to be true.
If both our assumptions hold, then building loyal AIs could serve as a useful intermediate step towards building aligned AIs. In particular, building loyal AIs might extend our ability to control powerful AI agents, even if these methods are unlikely to remain effective against vastly superhuman intelligence. The goal would not necessarily be to solve alignment for super intelligence, but rather to maintain control over AI systems long enough to leverage their capabilities to solve the harder alignment problems. For example, loyal AIs might prove to be more robust monitors of other AI systems, or more honest and effective researchers working on alignment itself.
AI Corrigibility
Perhaps the most relevant and well established subfield of AI safety to AI loyalty is corrigibility. For an introduction to the field, I recommend starting with Max Harms's CAST sequence. In addition to reading Max Harm's posts, I strongly recommend going through the Bibliography and Miscellaneous section of the linked post. The LW Wiki on Corrigibility is another valuable resource for exploring the existing literature on the subject.
There are several related definitions of corrigibility and even some disagreement over whether corrigibility constitutes a single, well defined trait. Broadly speaking, corrigibility refers to "the quality or state of being capable of being corrected, reformed, or managed." Different authors have expanded on this definition in different directions, but two particularly relevant features of corrigibility are: 1. Deference to a principal: Much like loyalty, corrigibility places the principal in a privileged position relative to the AI agent. Rather than attempting to instill morals directly into an AI agent, a corrigible agent defers to its principal's judgment and remains open to having its goals and behavior corrected. In this sense, both corrigibility and loyalty attempt to sidestep the problem of instilling the "right values" into an AI by instead relying on the values and judgment of its principal. 2. Caution and Low Impact: An important motivation behind corrigibility is that a corrigible agent should fail gracefully. Corrigible agents are cautious about taking actions that could undermine its principal's ability to correct it. This trait creates a motivation for the agent to avoid taking irreversible actions, preserve options for correction, and, where appropriate, defer to its principal rather than acting independently.
Corrigibility vs. Loyalty
The core similarity between corrigibility and loyalty is that both approaches prioritize making powerful AIs responsive to a particular principal rather than attempting to instill some independent conception of what is good. This, however, also means that both approaches potentially introduce concentration of power and AI misuse risks, since an AI that faithfully serves its principal may amplify that principal's power and execute its instructions even if the principle's objectives are malign or harmful.
The core difference between the two approaches are that whereas corrigibility emphasizes caution and openness to correction, loyalty emphasizes actively advancing the interests of the principal. This difference causes loyalty to interfere less with agency (a significant issue that plague corrigible agent approaches) but at the cost of sacrificing arguably the most attractive feature of corrigibility, namely, the possibility of graceful failure.
Secret Loyalties
A more recent but closely related active research agenda is the secret loyalties agenda. This line of research was motivated in part by Forethought's thought provoking article on AI-enabled coups, in which the authors explore the possibility that individuals or institutions could instill secret loyalties in AI models and subsequently exploit those loyalties to seize power. In this context, AI loyalty is framed primarily as a potential concentration of power threat.
There is substantial overlap between the AI loyalty and secret loyalties research agendas. Both require developing evaluations and metrics for measuring loyalty in AI systems, as well as methods for instilling loyalty in AI agents. However, there are two key differences: 1. The secret loyalties research agenda views emergent or instilled loyalties as harmful and a potential concentration of power threat, whereas the AI loyalty agenda is interested in investigating the benefits of instilling loyalties in AI agents, especially in mitigating loss of control. 2. The secret loyalties agenda focuses on undisclosed loyalties, particularly developing methods for monitoring and detecting loyalties that models may actively conceal. The AI loyalty agenda, on the other hand, is less concerned with concealment and more focused on training and measuring loyalty to an intended principal.
Other Related Areas
While looking into this subject I have also found literature outside the AI safety community which could be useful to the AI loyalty agenda.
AI Loyalty and Fiduciary AI
One especially relevant body of literature exists under the label fiduciary AI, and explores how AI systems can be designed to act in the best interests of their users.
With the growing interest in multi-agent systems and interactions involving multiple parties, researchers have also begun investigating how AI agents navigate competing instructions and interests.
Thank you very much for reading my post! This is by no means a comprehensive review, and I would greatly appreciate you sharing suggestions for additional papers or relevant research directions that I may have overlooked in the comments.
In this post, I attempt to provide a literature review of research related to AI loyalty.
In my previous post, I discussed and attempted to define AI loyalty as a distinct trait of AI agents. In this post, I aim to map the existing research landscape, highlighting the active areas of related research. By examining these connections, we can distinguish AI loyalty from related fields in AI Safety and identify a path forward for crafting this research agenda.
What is AI Loyalty?
Before diving into the literature, it is worth recapping what we mean by AI loyalty. We define an AI as loyal to a principle if:
1. The AI will follow the instructions of its principle more than it will follow the instructions of any other entity.
2. The AI is willing to incur greater costs, expend more effort, or make greater sacrifices on behalf of its principle than on behalf of any other entity.
Why Care About Loyalty? Theory of Victory
The usefulness of pursuing research on AI loyalty for AI safety hinges on two key assumptions:
1. Building loyal AIs is easier than building aligned AIs
2. AI loyalty reduces the risk of loss of control.
It is important to note that neither of these assumptions have yet been demonstrated to be true.
If both our assumptions hold, then building loyal AIs could serve as a useful intermediate step towards building aligned AIs. In particular, building loyal AIs might extend our ability to control powerful AI agents, even if these methods are unlikely to remain effective against vastly superhuman intelligence. The goal would not necessarily be to solve alignment for super intelligence, but rather to maintain control over AI systems long enough to leverage their capabilities to solve the harder alignment problems. For example, loyal AIs might prove to be more robust monitors of other AI systems, or more honest and effective researchers working on alignment itself.
AI Corrigibility
Perhaps the most relevant and well established subfield of AI safety to AI loyalty is corrigibility. For an introduction to the field, I recommend starting with Max Harms's CAST sequence. In addition to reading Max Harm's posts, I strongly recommend going through the Bibliography and Miscellaneous section of the linked post. The LW Wiki on Corrigibility is another valuable resource for exploring the existing literature on the subject.
There are several related definitions of corrigibility and even some disagreement over whether corrigibility constitutes a single, well defined trait. Broadly speaking, corrigibility refers to "the quality or state of being capable of being corrected, reformed, or managed." Different authors have expanded on this definition in different directions, but two particularly relevant features of corrigibility are:
1. Deference to a principal: Much like loyalty, corrigibility places the principal in a privileged position relative to the AI agent. Rather than attempting to instill morals directly into an AI agent, a corrigible agent defers to its principal's judgment and remains open to having its goals and behavior corrected. In this sense, both corrigibility and loyalty attempt to sidestep the problem of instilling the "right values" into an AI by instead relying on the values and judgment of its principal.
2. Caution and Low Impact: An important motivation behind corrigibility is that a corrigible agent should fail gracefully. Corrigible agents are cautious about taking actions that could undermine its principal's ability to correct it. This trait creates a motivation for the agent to avoid taking irreversible actions, preserve options for correction, and, where appropriate, defer to its principal rather than acting independently.
Corrigibility vs. Loyalty
The core similarity between corrigibility and loyalty is that both approaches prioritize making powerful AIs responsive to a particular principal rather than attempting to instill some independent conception of what is good. This, however, also means that both approaches potentially introduce concentration of power and AI misuse risks, since an AI that faithfully serves its principal may amplify that principal's power and execute its instructions even if the principle's objectives are malign or harmful.
The core difference between the two approaches are that whereas corrigibility emphasizes caution and openness to correction, loyalty emphasizes actively advancing the interests of the principal. This difference causes loyalty to interfere less with agency (a significant issue that plague corrigible agent approaches) but at the cost of sacrificing arguably the most attractive feature of corrigibility, namely, the possibility of graceful failure.
Secret Loyalties
A more recent but closely related active research agenda is the secret loyalties agenda. This line of research was motivated in part by Forethought's thought provoking article on AI-enabled coups, in which the authors explore the possibility that individuals or institutions could instill secret loyalties in AI models and subsequently exploit those loyalties to seize power. In this context, AI loyalty is framed primarily as a potential concentration of power threat.
Since this is a relatively new area of research, it has been difficult to find an established body of literature on the subject. I have found the work of Formation Research to be a useful starting point, particularly AIs with Secret Loyalties are a Serious but Addressable Threat and Narrow Secret Loyalty Dodges Black-Box Audits.
There is substantial overlap between the AI loyalty and secret loyalties research agendas. Both require developing evaluations and metrics for measuring loyalty in AI systems, as well as methods for instilling loyalty in AI agents. However, there are two key differences:
1. The secret loyalties research agenda views emergent or instilled loyalties as harmful and a potential concentration of power threat, whereas the AI loyalty agenda is interested in investigating the benefits of instilling loyalties in AI agents, especially in mitigating loss of control.
2. The secret loyalties agenda focuses on undisclosed loyalties, particularly developing methods for monitoring and detecting loyalties that models may actively conceal. The AI loyalty agenda, on the other hand, is less concerned with concealment and more focused on training and measuring loyalty to an intended principal.
Other Related Areas
While looking into this subject I have also found literature outside the AI safety community which could be useful to the AI loyalty agenda.
AI Loyalty and Fiduciary AI
One especially relevant body of literature exists under the label fiduciary AI, and explores how AI systems can be designed to act in the best interests of their users.
I particularly recommend Benthall and Shekman (2023), Designing Fiduciary Artificial Intelligence which offers a useful framework for assessing a principal's best interests.
Principal Loyalty in Multi-Party Interactions
With the growing interest in multi-agent systems and interactions involving multiple parties, researchers have also begun investigating how AI agents navigate competing instructions and interests.
One relevant paper is Li and Shi (2026), Whose Side Is Your Agent On? which examines questions of agent loyalty in multi-party settings.
Thank you very much for reading my post! This is by no means a comprehensive review, and I would greatly appreciate you sharing suggestions for additional papers or relevant research directions that I may have overlooked in the comments.