Marginal Risk is BS

David Scott Krueger (formerly: capybaralet)
Wed, 22 Apr 2026 05:20:10 GMT

I don’t know where the idea of “marginal risk” came from in AI policy. It sounds like BS. Yet another excuse to keep building dangerous AI systems…

The basic idea is that instead of looking at how likely your AI system is to lead to millions of deaths, you ask “given that other people are already building AI systems that might lead to millions of deaths, how much worse will I make things, if I build one more?” And then you only feel bad about that “marginal” contribution.

Does this exist in other areas? Imagine there are labs performing biological gain-of-function research (the kind that likely led to the COVID pandemic). Suppose your lab is doing a shit job of security and you estimate that every month, there’s a 1% chance that you cause another global pandemic. Now suppose that you learn that there is another lab that is doing an equally shit job of security. Does that make it OK, what you are doing? Obviously not.

So, when exactly are you allowed to say “The marginal risk that I might kill someone is small, because other people are also behaving recklessly. So I should be allowed to keep doing what I’m doing.” Not when they do things that actually kill people, generally!

So that’s the “Why aren’t we following accepted norms and standard practice?” objection. This applies to a lot of AI stuff, actually; see also: “evals” vs. engineering practices for other safety-critical technology -- It’s a world of difference.

There’s also the (closely related) moral objection: since when is “everyone else is doing it” an OK excuse? I talked about this previously; the answer is, in fact, “sometimes”, but I’d be surprised to find “when you are putting everyone’s lives at risk” in most people’s “acceptable” bucket.

Another closely related objection is that it’s anti-cooperative. Obviously the thing we should be doing is coordinating to not build any AIs that pose unacceptable risks. This isn’t a critique of individual actors making decisions this way, but of the normalization of a policy choice that leaves us with these unacceptable risks. What do you call such a policy? Unacceptable.

But I have a few more practical objections to considering marginal risk that I suspect some people might find more compelling.

First, there are only three to ten frontier AI developers, depending on how you count. So naively, the marginal risk from one developer should only be 3-10 times lower than the total risk. I don’t think a 3-10x reduction in the probability of human extinction, for instance, is likely to bring the current risk levels from “unacceptable” to “acceptable”. In practical terms, building one more frontier AI system gives us one more chance to mess it up and suffer the consequences. That’s non-trivial!

But also: Marginal risk can be reduced at every step of the way by moving at the same pace, but taking smaller steps. An increase in marginal risk that might seem too large can be reduced simply by inching towards it. If you were doing this unilaterally and didn’t have anybody else to compare it to, we could instead say you’re allowed a certain rate of increasing the marginal risk. But I haven’t seen anybody propose that.

Also, when we have N different AI developers, the effective rate of marginal risk increases overall gets multiplied by a factor of N. Let 10 companies increase the marginal risk by 1% each in a month, and you get a 10% increase. It’s basically a recipe for an incremental race-to-the-bottom.

Conclusion

Thinking “on the margin” is sometimes useful and appropriate, but there are good reasons we don’t usually do it for harming others, or risking harm. The way I’ve seen this phrase used in AI discourse is “reasonableness-washing” something that is actually quite silly. Marginal risk gives us baby steps towards catastrophe.

Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.

Share

Discuss