Last year, my colleagues on MIRI’s Technical Governance Team proposed an international agreement to halt risky development of superhuman artificial intelligence until it can be done safely. The agreement would require all clusters of AI chips with more computing power than 16 H100 GPUs to be registered with a coalition of states, led by the US and China, that would monitor their operations to ensure they aren’t being used for unsafe AI development. In my opinion, the proposal is impressively well-thought out and thorough – the authors identified various contingencies and closed many gaps in the plan.
However, one threat model was insufficiently addressed. Here is the agreement’s definition of clusters subject to registration requirements:
“Covered chip cluster (CCC) means any set of AI chips or networked cluster with aggregate effective computing capacity greater than 16 H100-equivalents. A networked cluster refers to chips that either are physically co-located, have inter-node aggregate bandwidth — defined as the sum of bandwidth between distinct hosts/chassis — greater than 25 Gbit/s, or are networked to perform workloads together. The aggregate effective computing capacity of 16 H100 chips is 15,840 TFLOP/s, or total processing power of 253,440 TFLOP-bit/s.”
Unfortunately, this definition had a sort of loophole: it leaves feasible the method of violating the agreement by doing distributed training, where instead of training models on a single large cluster with dedicated interconnect, the evader could conduct their training job on a decentralized network of small nodes, each calculating gradients on different subsets of the training data.
Note that the definition does explicitly include any set of hardware connected over a network to perform computing workloads together, so distributed training was already forbidden. But forbidden things aren’t necessarily enforceable. An evader could have collected thousands of GPUs in secret, into small nodes each individually below the reporting threshold, and if one were discovered, they could deny that they were doing anything prohibited.
MIRI has had this possibility in mind for a long time. When writing the agreement, the authors wanted to make the restrictions lenient enough that ordinary individuals would never violate them by accident, but still tight enough that unmonitored clusters can’t be networked together to train a dangerous model. They selected 25 gigabits per second, which is slower than AI hardware interconnect but faster than almost any internet connection to homes or facilities other than datacenters. Model developers typically rely on much faster connections to transfer large amounts of data, with consumer-grade internet connections considered impractically slow for model training.
Recently, I decided to investigate this possibility in more depth. I conducted a literature review on distributed training, and built a simulator that models the efficiency of training networks with many nodes, hierarchical configurations, and pipeline parallelism over the internet. The simulator takes the scaling behavior observed in published ML experiments and extrapolates it to predict the quality of models trained using distributed training at larger scales not yet attempted.
Based on the results of this research, we’ve made a simple change to the covered chip cluster definition that closes the distributed training loophole: any clusters with more than 1,280 GB of high-bandwidth memory are now also covered, and therefore subject to the registration and monitoring requirements. The new definition (with changes shown in bold) is as follows:
“Covered chip cluster (CCC) means any set of AI chips or networked cluster with aggregate effective computing capacity or accelerator memory greater than 16 H100-equivalents. A networked cluster refers to chips that either are physically co-located, have inter-node aggregate bandwidth — defined as the sum of bandwidth between distinct hosts/chassis — greater than 25 Gbit/s, or are networked to perform workloads together. The aggregate effective computing capacity of 16 H100 chips is 15,840 TFLOP/s, or total processing power of 253,440 TFLOP-bit/s. The total accelerator memory of 16 H100 chips is 1,280 GB.”
This simple change nearly doubles the cost for evaders to secretly train frontier-scale models using distributed computing clusters of unregistered hardware, and forces them to use five times as many nodes, since fewer chips per node are allowed under the memory limit. This vastly increases the amount of financial manipulation and operational secrecy that an evader would need for such a training run, and makes it much more detectable by countries party to the agreement.
Other mechanisms by which a secret distributed training network would be discovered include:
Initial chip consolidation: upon the start date of the agreement, parties would locate and begin tracking a very large percentage of existing AI chips, which leaves little room for an evader to squirrel away some chips that are not accounted for.
Whistleblowers: the treaty provides incentives and safe harbor for researchers who report illegal, secret operations. The distributed training setup would require tens of thousands of nodes, with technicians visiting each one for setup and maintenance. It would be impossible to guarantee that none of them would report the network, and keep every site hidden from foreign intelligence agencies.
Challenge inspections: Executive Council members (that is, the US and China) could do random inspections, or inspections of specific sites based on a party’s suspicion of such a distributed training operation. Though an evader can make it difficult using VPNs, bandwidth structuring, and additional misleading traffic, the inspections may find evidence linking one node to the rest of the network and revealing the whole operation.
Together, the methods above make it highly unlikely that an evader could keep such an operation secret.
AI is advancing rapidly, and we need worldwide cooperation to solve the risks. MIRI’s international agreement facilitates stable, reliable compute governance, preventing defectors from cheating and starting a dangerous race to the bottom on safety. An agreement without loopholes, that all parties trust to keep the world safe, is more likely to be adopted and is one of humanity’s best options for navigating the dangers of advanced artificial intelligence.
This blog post summarizes a forthcoming research report. Contact us if you would like to discuss the findings prior to publication.
Many thanks to Jaime Sevilla for his article about scaling decentralized training over the internet, and to Aaron Scher for several critical reviews that sharpened the project and directed its focus toward the issues most important to compute governance.
Last year, my colleagues on MIRI’s Technical Governance Team proposed an international agreement to halt risky development of superhuman artificial intelligence until it can be done safely. The agreement would require all clusters of AI chips with more computing power than 16 H100 GPUs to be registered with a coalition of states, led by the US and China, that would monitor their operations to ensure they aren’t being used for unsafe AI development. In my opinion, the proposal is impressively well-thought out and thorough – the authors identified various contingencies and closed many gaps in the plan.
However, one threat model was insufficiently addressed. Here is the agreement’s definition of clusters subject to registration requirements:
Unfortunately, this definition had a sort of loophole: it leaves feasible the method of violating the agreement by doing distributed training, where instead of training models on a single large cluster with dedicated interconnect, the evader could conduct their training job on a decentralized network of small nodes, each calculating gradients on different subsets of the training data.
Note that the definition does explicitly include any set of hardware connected over a network to perform computing workloads together, so distributed training was already forbidden. But forbidden things aren’t necessarily enforceable. An evader could have collected thousands of GPUs in secret, into small nodes each individually below the reporting threshold, and if one were discovered, they could deny that they were doing anything prohibited.
MIRI has had this possibility in mind for a long time. When writing the agreement, the authors wanted to make the restrictions lenient enough that ordinary individuals would never violate them by accident, but still tight enough that unmonitored clusters can’t be networked together to train a dangerous model. They selected 25 gigabits per second, which is slower than AI hardware interconnect but faster than almost any internet connection to homes or facilities other than datacenters. Model developers typically rely on much faster connections to transfer large amounts of data, with consumer-grade internet connections considered impractically slow for model training.
Recently, I decided to investigate this possibility in more depth. I conducted a literature review on distributed training, and built a simulator that models the efficiency of training networks with many nodes, hierarchical configurations, and pipeline parallelism over the internet. The simulator takes the scaling behavior observed in published ML experiments and extrapolates it to predict the quality of models trained using distributed training at larger scales not yet attempted.
Based on the results of this research, we’ve made a simple change to the covered chip cluster definition that closes the distributed training loophole: any clusters with more than 1,280 GB of high-bandwidth memory are now also covered, and therefore subject to the registration and monitoring requirements. The new definition (with changes shown in bold) is as follows:
“Covered chip cluster (CCC) means any set of AI chips or networked cluster with aggregate effective computing capacity or accelerator memory greater than 16 H100-equivalents. A networked cluster refers to chips that either are physically co-located, have inter-node aggregate bandwidth — defined as the sum of bandwidth between distinct hosts/chassis — greater than 25 Gbit/s, or are networked to perform workloads together. The aggregate effective computing capacity of 16 H100 chips is 15,840 TFLOP/s, or total processing power of 253,440 TFLOP-bit/s. The total accelerator memory of 16 H100 chips is 1,280 GB.”
This simple change nearly doubles the cost for evaders to secretly train frontier-scale models using distributed computing clusters of unregistered hardware, and forces them to use five times as many nodes, since fewer chips per node are allowed under the memory limit. This vastly increases the amount of financial manipulation and operational secrecy that an evader would need for such a training run, and makes it much more detectable by countries party to the agreement.
Other mechanisms by which a secret distributed training network would be discovered include:
Together, the methods above make it highly unlikely that an evader could keep such an operation secret.
AI is advancing rapidly, and we need worldwide cooperation to solve the risks. MIRI’s international agreement facilitates stable, reliable compute governance, preventing defectors from cheating and starting a dangerous race to the bottom on safety. An agreement without loopholes, that all parties trust to keep the world safe, is more likely to be adopted and is one of humanity’s best options for navigating the dangers of advanced artificial intelligence.
This blog post summarizes a forthcoming research report. Contact us if you would like to discuss the findings prior to publication.
Many thanks to Jaime Sevilla for his article about scaling decentralized training over the internet, and to Aaron Scher for several critical reviews that sharpened the project and directed its focus toward the issues most important to compute governance.