The Problem with Defining an "AGI Ban" by Outcome (a lawyer's take).
TL;DR Some “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That’s legally insufficient: regulation has to act before harm occurs, not after. * Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same. * Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. * Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, or systematic deception (these are merely illustrative examples, not formal proposals). * Bring lawyers in. If this definitional work is left to policymakers, they’ll default to blunt proxies (compute caps). In-house counsels will then map the gray zones and advise on how to bypass the ban. Bottom line: If we want a credible AGI ban, we must define and prohibit precursor capabilities with the same precision and enforceability that nuclear treaties applied to fissile material. Anything less collapses into Goodharting and fines-as-business-cost. Why outcome-based AGI bans' proposals don’t work I keep seeing proposals framed as “international bans on AGI,” where AGI is defined extremely broadly, often something like “whatever AI companies develop that could lead to human extinction.” As a lawyer, I can’t overstate how badly this type of definition fails to accomplish its purpose. To enable a successful ban on AGI, regulation has to operate ex ante: before the harm materialises. If the prohibited category is define
This was a great read! I swear I've watched The Notebook at least twice over the years and, for the life of me, I can't remember that scene.
I would be very curious to hear your take on 500 Days of Summer. Do you think it portrays a better or more realistic picture on how these sort of unreciprocated pursuits tend to go? I really enjoyed that in the end the person that kept saying that she just doesn't believe in marriage married somebody else. I think that was one of my favorite details of the movie.