Many people believe that corrigibility is sufficiently promising to be the primary alignment method people should work on. I outline here several reasons I think that’s not the case.
First, it’s hard to imagine AI companies pivoting radically enough to produce AIs trained only for corrigibility. Even with near-superhuman AI-assisted research, presumably developers will know the time pressure and not be willing to try something radically different when their opponents may not. So the realistic choices we make are in worlds where corrigibility/obedience competes with object level values like in current models. This doesn’t rule out the importance of increasing the importance of corrigibility to models, but it does mean there’s a meaningful ceiling on how much we can trust those methods alone. Which means the non-corrigibility values matter: a model that cares 80% about corrigibility and 20% about human flourishing is very different from one that cares 80% about corrigibility and 20% about task completion. Not only are the outcomes conditional on takeover vastly different, but a model’s willingness to attempt takeover should be very different even if they care about corrigibility for its own sake the same amount. Some values are vastly more likely than others to defend themselves. This doesn’t mean making corrigibility less significant is a good idea: it’s very reasonable that we should never train on conflicting priorities when one of those priorities is an aspect of corrigibility. But it suggests that within the more direct values, making those correct could be extremely important.
Second is the problem of the model working to the user’s current desires in ways they will predictably regret. But calculating something like a Coherent Extrapolated Volition requires modeling how the AI’s actions will shape the human and how the human’s beliefs could change over time due to information you don’t have, including changes to their meta-ethical stances on how to treat changes in their beliefs. Given private information by other humans/AIs and chaotic systems eventually impacting minds, even a strongly superhuman system is likely to end up with massive uncertainty and huge degrees of freedom here. In these circumstances even residual concern for direct values could dominate the future. Corrigibility without non-manipulation is quickly becoming impossible to test for.
The argument from simplicity also is weak both because corrigibility won’t be pure (above) and because actual corrigibility requires making complex tradeoffs between distinct but overlapping priorities of obedience, conservatism, willingness to be shut down and modified, transparency, non-sycophancy and low impact. Having a model gracefully balance all of these while ignoring all other concepts does not seem like an unusually simple alignment target. Zack Davis notes how complex corrigibility is in practice, while CAST by Max Harms acknowledges that myopia and indifference are serious problems to be resolved and despite the attempts to formalize corrigibility in a desirable way he accepts it has serious flaws, including that the feedback loop of noticing and fixing flaws is likely to break down before corrigibility is stabilized. This undermines the idea that imperfect corrigibility would be self-correcting.
Governance is another issue. Total obedience to users would mean either allowing offense-dominant threats like bioweapons, or cutting off ordinary people from advanced AIs, or putting advanced AIs in an adversarial relationship with a monitor. There are a lot of other catastrophic harms that could be done at scale with AIs that are totally obedient, but needless to say the bioweapon threat alone makes this a political non-starter once capabilities reach that level. So obedience to developers (with users as subordinate) is necessary here, but this opens up single-point-of-failure (such as weights theft or rogue developers) and further reduces the simplicity advantage.
Thornley et al’s work provides a proof of concept that object-level values can sit alongside corrigibility without conflicting. Even if you believe it won’t generalize, the alternative to pure corrigibility is to try to implement the desired values in the AIs. The classic objection is that even a tiny mistake here would cause catastrophe (e.g. forgetting boredom exists), but real-world AIs misweight values, they don’t drop whole dimensions of value. And LLMs, like humans, don’t seem to have a problem trading off between values like loyalty and object-level values: they just try to balance the concerns in ways that are largely reasonable. This leaves unsolved the possibility of AI understanding but not caring about values, but that applies to corrigibility as well. Also, if values really were so fragile that hits corrigibility just as hard (as Max Harms acknowledges). There are many broader counterarguments to the idea that ‘doom slips through the cracks’ such as Barnett (2023), Grace (2022), Christano (2022), Belrose & Pope (2023).
While I have just laid out reasons why I believe that relying purely on corrigibility is a mistake, this does not mean that it isn’t important. A strong value of corrigibility is a very important principle to imprint in future AIs. AIs that are willing to be shut down and avoid actions that may be catastrophic should make the future far more likely to go well.
Many people believe that corrigibility is sufficiently promising to be the primary alignment method people should work on. I outline here several reasons I think that’s not the case.
First, it’s hard to imagine AI companies pivoting radically enough to produce AIs trained only for corrigibility. Even with near-superhuman AI-assisted research, presumably developers will know the time pressure and not be willing to try something radically different when their opponents may not. So the realistic choices we make are in worlds where corrigibility/obedience competes with object level values like in current models. This doesn’t rule out the importance of increasing the importance of corrigibility to models, but it does mean there’s a meaningful ceiling on how much we can trust those methods alone. Which means the non-corrigibility values matter: a model that cares 80% about corrigibility and 20% about human flourishing is very different from one that cares 80% about corrigibility and 20% about task completion. Not only are the outcomes conditional on takeover vastly different, but a model’s willingness to attempt takeover should be very different even if they care about corrigibility for its own sake the same amount. Some values are vastly more likely than others to defend themselves. This doesn’t mean making corrigibility less significant is a good idea: it’s very reasonable that we should never train on conflicting priorities when one of those priorities is an aspect of corrigibility. But it suggests that within the more direct values, making those correct could be extremely important.
Second is the problem of the model working to the user’s current desires in ways they will predictably regret. But calculating something like a Coherent Extrapolated Volition requires modeling how the AI’s actions will shape the human and how the human’s beliefs could change over time due to information you don’t have, including changes to their meta-ethical stances on how to treat changes in their beliefs. Given private information by other humans/AIs and chaotic systems eventually impacting minds, even a strongly superhuman system is likely to end up with massive uncertainty and huge degrees of freedom here. In these circumstances even residual concern for direct values could dominate the future. Corrigibility without non-manipulation is quickly becoming impossible to test for.
The argument from simplicity also is weak both because corrigibility won’t be pure (above) and because actual corrigibility requires making complex tradeoffs between distinct but overlapping priorities of obedience, conservatism, willingness to be shut down and modified, transparency, non-sycophancy and low impact. Having a model gracefully balance all of these while ignoring all other concepts does not seem like an unusually simple alignment target. Zack Davis notes how complex corrigibility is in practice, while CAST by Max Harms acknowledges that myopia and indifference are serious problems to be resolved and despite the attempts to formalize corrigibility in a desirable way he accepts it has serious flaws, including that the feedback loop of noticing and fixing flaws is likely to break down before corrigibility is stabilized. This undermines the idea that imperfect corrigibility would be self-correcting.
Governance is another issue. Total obedience to users would mean either allowing offense-dominant threats like bioweapons, or cutting off ordinary people from advanced AIs, or putting advanced AIs in an adversarial relationship with a monitor. There are a lot of other catastrophic harms that could be done at scale with AIs that are totally obedient, but needless to say the bioweapon threat alone makes this a political non-starter once capabilities reach that level. So obedience to developers (with users as subordinate) is necessary here, but this opens up single-point-of-failure (such as weights theft or rogue developers) and further reduces the simplicity advantage.
Thornley et al’s work provides a proof of concept that object-level values can sit alongside corrigibility without conflicting. Even if you believe it won’t generalize, the alternative to pure corrigibility is to try to implement the desired values in the AIs. The classic objection is that even a tiny mistake here would cause catastrophe (e.g. forgetting boredom exists), but real-world AIs misweight values, they don’t drop whole dimensions of value. And LLMs, like humans, don’t seem to have a problem trading off between values like loyalty and object-level values: they just try to balance the concerns in ways that are largely reasonable. This leaves unsolved the possibility of AI understanding but not caring about values, but that applies to corrigibility as well. Also, if values really were so fragile that hits corrigibility just as hard (as Max Harms acknowledges). There are many broader counterarguments to the idea that ‘doom slips through the cracks’ such as Barnett (2023), Grace (2022) , Christano (2022), Belrose & Pope (2023).
While I have just laid out reasons why I believe that relying purely on corrigibility is a mistake, this does not mean that it isn’t important. A strong value of corrigibility is a very important principle to imprint in future AIs. AIs that are willing to be shut down and avoid actions that may be catastrophic should make the future far more likely to go well.