This post examines AI alignment through the lens of systems thinking and safety engineering. We aim to identify structural mechanisms that can maintain alignment in complex sociotechnical systems, systems where AIs interact with multiple human operators and stakeholders.[1] One conception of AI misalignment is a control problem where the behavior...
In 1984, Charles Perrow wrote the book Normal Accidents: Living with High-Risk Technologies. The book is an examination of the causes of accidents and disasters in highly complex, technological systems. In our modern time it can be helpful to review the lessons that Perrow set forth, as there may be...
This post examines AI alignment through the lens of systems thinking and safety engineering. We aim to identify structural mechanisms that can maintain alignment in complex sociotechnical systems, systems where AIs interact with multiple human operators and stakeholders.[1]
One conception of AI misalignment is a control problem where the behavior of an AI system diverges from safety constraints and governing principles. Unlike simple human-AI pairs, deployed AI systems operate in hierarchical sociotechnical environments. These environments involve multiple operators or stakeholders with competing objectives. Therefore, understanding and preventing misalignment requires analysis at the system level, scrutiny of individual components is insufficient.
The framework presented here draws on established principles from safety engineering and systems analysis:
In 1984, Charles Perrow wrote the book Normal Accidents: Living with High-Risk Technologies. The book is an examination of the causes of accidents and disasters in highly complex, technological systems. In our modern time it can be helpful to review the lessons that Perrow set forth, as there may be no technology with more potential risks than AI.
I’m going to review some of the concepts that Perrow covered in this book, and in his other book The Next Catastrophe. I’ll explore how they might be used to construct safer AI systems, less prone to loss of control or pursuing undesired objectives, as well as misuse. I’ll also argue that making smaller, specialized... (read 3237 more words →)