I’ve heard a number of people say that it’s unclear what the technical contours of a global AI treaty would look like. That is true - but it’s not actually an obstacle to negotiating an international treaty.
I agree with the overall point of this post, but, re: the bolded section: I wanna remind everyone that MIRI has put forth an example treaty proposal. (There are a couple versions and I'm not sure what's the most up-to-date working one).
https://ifanyonebuildsit.com/treaty
If you think those don't count for some reason think there should at least be a footnote explaining why.
It's not that they don't count, it's that they won't go anywhere, and so aren't going to dictate the actual contents of any actual future treaty. As I said later in the post:
Other proposals are actually trying to solve the problem directly. For example, the intentional equivalent of the ill-fated 1946-proposed Baruch Plan... Because directly solving the problem by fiat, imposed globally, before negotiations between parties start, is very hard and not usually effective.
There are also many papers on various parts of governance solutions, such as Building The Pause Button: https://arxiv.org/abs/2506.20530
Strong agree. I think pretty much all treaties where a party has to give something up start out as some high-level agreement on a common goal, with details left to work out during the actual negotiations.
Take free trade agreements for example. Two (or more) countries start from a high-level agreement in principle that they want to increase trade between their countries. But each side will have certain industries or domestic standards that they wish to protect, which are incompatible with the other side's goal of increased market access. Each will go in with an idea of what they are willing to give up, and each will have their own "red lines". Neither side knows going in what the final rules will be. During the negotiation process, information is shared, problems are identified, solutions are proposed, and compromises are made. Sometimes it turns out after several rounds that the two sides' red lines are fundamentally incompatible, and negotiations fail. But treaties where countries agree to give up something significant never start with all the details ironed out.
A related objection I often hear is "China will never agree to that" or "the US will never agree to that". We don't know that. They may not like a particular proposal, but they may still agree to it if they get a bunch of other concessions in the treaty. We have no idea what China or the US's red lines are yet, so such objections are grossly premature.
I’ve heard a number of people say that it’s unclear what the technical contours of a global AI treaty would look like. That is true - but it’s not actually an obstacle to negotiating an international treaty.
I’ll try to explain why this isn’t a good objection, but the short version is that if countries have clear goals which are largely shared, negotiations end up with strong treaties. So the important questions are not the exact rules. The critical questions are if there really is a joint global risk that requires action - and experts agree there is, and whether verification and enforceability are possible - and experts say they are. So the problem isn’t a technical issue, it’s a question of whether we can get to an agreement. And despite facile “we can’t stop until they do” arguments, we can and should try to do better.
In order to explain why we do not need to figure out the details first, it's worth talking about other treaties.
The Pandemic Treaty (Task Failed Successfully)
I will start with the example I watched most closely, over the past five years. The Pandemic treaty was proposed in 2021, “when WHO member States agreed on the urgent need for a legally binding international instrument,” per the UN. It was supposed to fix all the problems we had during COVID. Unfortunately, this didn’t include preventing pandemics, and past that point, no-one agrees on what things should have been done, or what to do next time. So, if we can’t agree on anything, what does the treaty do? Mostly, generic pandemic-stopping stuff - “commitment to a ‘One Health’ approach to pandemic prevention, stronger national health systems, setting up a coordinating financial mechanism, and creating a globally coordinated supply chain and logistics network for health emergencies.”
How much of that was agreed about in mid-2021 when it was proposed by the European Council President? Basically none of it. What are the actual commitments? Well, that’s complicated, but the short version is that there aren’t any. The treaty insists that countries get to stay in control of their national health systems, and no-one could tell them what to do, or how - which sounds a lot like the failure that allowed COVID-19 to spread. Lots of people had different visions for what the treaty should do, from global vaccine justice to enhancing global public health to providing funds for response to climate change issues to supply chain resilience to considering animal and plant health in pandemic planning, and it ended up as a mishmash of random things that people proposed. But the failure here was one of vision - it was unclear what the world would get out of a treaty that everyone agreed about, and that lack was never addressed.
That’s obviously a problem, but not the central one we have with proposing a global treaty to ban unsafe ASI. For those asking for such a treaty, we agree about what needs to be stopped, namely, building unsafe ASI. The questions are all about how to make that happen. So we should look at another example, and I think the best parallel is nuclear weapons.
The Nuclear Treaty
If you know anything about the history of nuclear weapons treaties, you read the section title and immediately asked: “which one?” And there are so many options - there was the Limited test ban treaty in 1963, the treaty on the Non-Proliferation of Nuclear Weapons (NPT) in 1968, the Strategic Arms Limitation Treaty (SALT) in 1972, the Anti-Ballistic Missile (ABM) treaty in 1974, the Convention on Physical Protection of Nuclear Materials (CPPNM) in 1980, and the New START Treaty in 2010. And that’s ignoring almost a half-dozen regional “Nuclear Weapon-Free Zone” treaties.
Why were there so many treaties? The goal - preventing use of nuclear weapons - was broadly agreed upon. But the exact way to accomplish that goal was tricky. And that led to lots of uncertainty, and the need for a number of different treaties addressing different parts of the problem, from testing to proliferation - but the goal was a north star. That meant that every time a concern arose, countries tried hard to figure out what was needed to accomplish the goal of not having a nuclear war, and what rules move the world further from that outcome. Even outside of treaties, nuclear powers generally have embraced a no-first-use doctrine, and have taken other measures to reduce the risk of accidental escalation. All of these address the messy dynamics of nuclear escalation between states that can’t risk being left behind.
What does this tell us about treaties about AI risks?
Lessons for Possible AI Treaties
There are a dozen suggested international treaties for AI, and most aren’t what I’m discussing. Many are the equivalent of the 1959 Antarctic Treaty, which was the first nuclear-relevant treaty, but which only said countries aren’t allowed to use Antarctica to test nuclear weapons or dispose of waste. In my view, the analogous proposals include any treaty that does anything about AI other than making sure future systems do not end cause global catastrophes. (That doesn’t mean such treaties are a bad idea, just that they aren’t what I’m discussing here.)
Other proposals are actually trying to solve the problem directly. For example, the intentional equivalent of the ill-fated 1946-proposed Baruch Plan, which tried to put all nuclear weapons under international control. And the Soviet counterproposal agreed about the goal - prevention of the production and use of nuclear weapons - but disagreed about how it should work. Which meant there was no agreement to stop proliferation for several decades. Because directly solving the problem by fiat, imposed globally, before negotiations between parties start, is very hard and not usually effective.
Luckily, for nuclear treaties it didn’t matter. The overwhelming consensus of the public was that we shouldn’t have a nuclear war, and despite claims that it was an inevitable race that could only end with disaster, nuclear weapons haven’t been used in 80 years and counting. The specifics of the treaties were critical, and navigating to success was in fact hard, without any final victory. As noted above, these are a mix of bilateral, multilateral, and global treaties. Of course, we’re still worried about nuclear proliferation and nuclear war. But the treaties have given the world enough stability to avoid disaster, so far.
Again, the world was lucky that building nuclear weapons went slowly, for two decades, and that the norm against use got established in the run-up to treaties. But no-one should argue that the lack of nuclear war means the treaties were unnecessary - if anything, the common argument is that the treaties are too weak and contingent.
Do we need an ASI treaty? Yes - and that is true even if you think the risk is minimal. Yes, there is debate about whether the risk will be realized, but there is clear consensus that it is a risk, and that we should have at least the ability to put rules in place. Should we have one grand treaty, or plan for solving one part at a time? It’s unclear. Many treaty areas have dozens of overlapping treaties; “the” Geneva convention is a series of treaties that added clarifications and rules over decades. But we do need to get started. People who think we’re a decade away from fully general AI should be terrified that it’s already late in the game to start discussing such a treaty. And people who are saying we might have ASI by the end of the decade are mostly already screaming about the need for some international rules.
Do Treaties Solve the Problem? (Do We Need Other Rules?)
Nuclear power is regulated, partially via an international body - that’s mostly not about preventing global thermonuclear war, but it's critically related due to the need to control nuclear materials. And at a national level, nuclear medicine is regulated, radiation exposure levels are regulated, and we have tons of other rules that don’t relate to preventing a Nuclear World War Three. That doesn’t mean those rules aren’t important.
For AI, we already need regulation on various uses and misuses. Some of these might be international - bans on autonomous weapons, bans of nonconsensual pornography, bans on use of AI to violate other international laws. Others should be national, or even local, like bans of deceptive use of AI, or use of AI to break other laws. That’s how regulation works.
But these are not the AI treaties that we need, they are just places where we need rules. And they should be pursued - just not at the cost of further delay on the global catastrophic risks we face.
AI NotKillEveryoneism Treaties
To have effective treaties that stop AI-driven global catastrophes, there are many questions that need to be answered. What exactly is needed to prevent the creation of misaligned ASI? Which chips should be tracked, who should be allowed to use them, and for what types of model development? What controls or safety measures must be in place? How should we measure dangerous capabilities? Where should the line be? Which countries need to agree first? How can we manage multilateral coordination when only a few countries are leading in the race? Will we ban existing frontier systems at the time the treaty comes into force? Or will groups be able to build slightly more capable systems with additional safeguards? What should those safeguards be?
Fortunately, we don’t need final answers to all of these questions in order to start negotiating a treaty. We don’t even need final answers in order to sign a treaty - many treaties have mechanisms for routine updates of rules. But we need a clear vision - no-one builds ASI until we are sure it is safe, and when people try clever ways to get around whatever rules exist, they are told to stop, and told that the rules will be updated. Companies must be told that their race to ASI is over, that the risk is too high, and no-one gets to win. Countries are told that whatever geopolitical advantage they think they will gain from building ASI isn’t allowed, because winning a race without clear safety rules is overwhelmingly likely to kill us all.
Will this be a single treaty, or require several over time? I don’t know. It also doesn’t matter. But it needs to start now, because international coordination takes time and is slow. We do not need to specify the outcome before starting, and uncertainty isn’t a reason to wait to build momentum and putting pieces in place so we can discuss what is possible to restrict or ban, how it will be verified, and howand why to ensure participants will prefer compliance to defection. Lock-in on the details is a risk, but waiting until we need immediate action isn’t a way to make the eventual responses better - quite the opposite, since delay eliminates capacity to explore the details. Obviously, work on a framework treaty needed to start at least a few years ago when the risks became globally clear, and we can hope we aren’t too late given the clear imminent risks of superintelligence.
In short, saying that we can’t discuss a treaty yet because we don’t know what the rules need to be is an historically illiterate and poorly reasoned objection.