I’ve noted before that it can be difficult to separate progress in safety from progress in capabilities. However, doing so is important, as we want to ensure that we are making differential progress on safety, rather than just advancing safety as a consequence of advancing capabilities. In particular, I think that research should rigorously evaluate trade-offs between improving safety and advancing capabilities.

This paper introduces the concept of “capabilities externalities,” and provides concrete steps for how to analyze the risk of different research papers. It also introduces “x-risk sheets,” which can be added to papers and help researchers discuss their impact on AI x-risk. Since discussion of AGI is becoming increasingly normalized in mainstream ML, I think careful discussions of x-risk are increasingly feasible within mainstream ML. The paper is largely aimed at mainstream ML researchers, and it is one step towards normalizing the discussion of existential risks.