Our key insight is a pessimistic one: this is the sort of situation which, though individuals and markets don’t handle it well, isn’t actually handled well by governments either. The fundamental mistake of statist thinking is to juxtapose the tragically, inevitably flawed response of individuals and markets to large collective-action problems like this one against the hypothetical perfection of idealized government action, without coping with the reality that government action is also tragically and inevitably flawed.
Or, more simply, most statists, including those here, are fantasists. I strongly recommend the people here who have actually been advocating the regulation of AI research, for example, to read Hayek's The Fatal Conceit, with a view as how politicians and bureaucrats, those who would actually implement such a regulation, would really use it, rather than your fantasies about how it should be enforced.
And that is entirely beside the fact that it would require a world-wide totalitarian dictatorship to enforce.
Well, as Robin Hanson said, coordination is hard.
If it's worth saying, but not worth its own post, even in Discussion, it goes here.