Why AI development won’t stop and what can realistically be done about it.
If war is so terrible, why do we still actively wage it? If nuclear weapons are so powerful and dangerous that we even came up with the concept of Mutually Assured Destruction (MAD) as a consequence of using them, why do we still keep stockpiles of them? If climate change is a scientifically proven fact whose consequences are not only awful, but global-reaching, why haven’t we successfully addressed it yet? If hunger could have already been eradicated, why haven’t we done so? If many of the most renowned AI scientists have warned against AI’s reckless development, why don’t we pause its development until at least we feel more certain about how to do it safely?
These questions are complex and difficult to address in the world we live in. Or in other words: if these problems were easy to manage, they would already have been solved.
In this post, I am making the claim that two fundamental challenges stand in the way of global coordination: 1) the visibility problem, concerning our ability to understand and agree on the nature and consequences of a threat; and 2) the structural problem, concerning the incentives that drive states to act even against the collective interests of individuals.
It seems to me that most of the current discussion around AI global coordination is missing out on either or both of these problems. Moreover, addressing either in isolation won't be sufficient.
Understanding these two problems helps explain why simply calling for an AI pause is unlikely to succeed—and what a more realistic path toward global coordination might look like. The result is a highly pragmatic (and speculative too, to be fair) take on why we can’t just pause AI development due to the visibility and structural issues and a proposal of the kind of solutions I think are necessary in order to even have a chance at global coordination of the sort needed.
The visibility problem
All states affected by a global challenge need to recognize it as both real and global. They also need to see themselves as sufficiently responsible for addressing it to take action. If they are expected to contribute equally, they will probably also need to perceive the challenge as similarly serious. Otherwise, they will not feel compelled to make comparable contributions.
The structural problem
States operate within an anarchic international order. There is no authority above states that can enforce laws and treaties, even when mutually interested states have signed them. Moreover, states ultimately pursue their own survival. They may care about the well-being of neighboring states, or even all states, but their primary responsibility lies within their own borders, and their survival comes first. States must also contend with substantial uncertainty about other states’ intentions. This uncertainty is a complex product of anarchy, states’ ability to wage war, their pursuit of survival, and other factors. Nonetheless, states cooperate and compete in this environment, both economically and militarily. They recognize that survival is best secured by being strong relative to their competitors. This is Realism 101. Within international relations scholarship, there is plenty of nuance and disagreement, for example about how much domestic affairs and international institutions affect states’ behavior. Without going too deeply into theory, however, anarchy, the pursuit of survival, and uncertainty create structural conditions that affect states’ decisions.
When confronted with a global issue, a state must evaluate possible outcomes through both its own perception of the problem and the structural constraints it faces. In academia, this is perhaps best captured by the security dilemma: one state’s decision to arm itself may look like preparation for aggression to its neighbor, even if its intentions are defensive. The neighbor is then motivated to arm itself in response. This creates a competitive spiral in which neither state may seek confrontation, yet both act according to the structure of the international order. The issue extends beyond military power and appears in many kinds of cooperative initiatives. Cooperation on a particular issue is always assessed through a game-theoretic lens: “What do I gain or lose relative to my competitors?” This occurs in many, if not all, areas of international relations, from climate cooperation to support for political movements, and from international aid to aggression.
The two problems together
Think of the structural problem as the rules of the game and the visibility problem as the players’ ability to see what lies ahead.
AI governance is possibly one of the most consequential global challenges we have ever faced. But before turning to it, let’s briefly consider two other examples.
Climate change
Climate change has posed an important visibility challenge. What would the consequences of inaction be? A decision-maker cannot answer only in general terms: how exactly would inaction affect their state in the years to come? Will it rain more, and by how much? What about heat waves? What consequences should they expect? What specific action would reduce a particular risk by a given amount? How much money should they allocate, and to what or whom? These are real questions that decision-makers have faced when deciding how to respond to climate change, and they are difficult to answer. Climate change illustrates the visibility problem: its effects unfold over years and decades, affect different parts of the world unequally, and involve consequences that are not clearly defined or agreed upon.
From a structural perspective, the problem becomes even muddier. Should all states contribute equally when responsibility for the problem is so unevenly distributed? If the consequences affect some states more than others, what ensures that less affected states also cooperate, or contribute as much? If reducing my state’s CO₂ emissions means slowing the development that earlier industrialized states benefited from, why should I now have to slow my own progress? How will climate cooperation affect my position relative to competitors in the years to come? Will I be better or worse off?
Those questions have structural reasons behind them and even if states see the challenge as real, they may still strategically “defect” as they see the benefits of doing so, relative to other players, as greater.
Nuclear weapons proliferation
A very different example is nuclear weapons. From a state’s decision-making perspective, nuclear weapons differ fundamentally from climate change: their visibility is crystal clear. Their capabilities and consequences were made vivid by the bombings of Hiroshima and Nagasaki and by the many nuclear tests that followed. Through this clarity, nuclear weapons became the most effective deterrent. It became clear what the consequences of attacking a nuclear-armed state would be.
This visibility also created a powerful structural incentive for other states to pursue nuclear weapons. The security dilemma associated with nuclear proliferation was particularly vivid during the Cold War. Once the United States had nuclear weapons, the Soviet Union felt threatened and pressured to develop its own. Once both had them, each felt it needed more weapons to ensure that it could retaliate with equal or greater force if attacked. This led to a cycle of nuclear armament in which the destruction of the other side could be assured, giving rise to the concept of mutually assured destruction.
The reason the nuclear standoff never escalated into war, despite its close calls, was that all parties could clearly see the consequences. Unlike climate change, nuclear proliferation was “blessed” with immense visibility for the states involved. Had the consequences been less clear, nuclear weapons would likely have been used. I have simplified the history considerably for the sake of argument, but broadly speaking, this is the main reason nuclear weapons have not been used again in war.
AI governance
In his September 2026 post We Must Pace the Frontier, Anthropic CEO Dario Amodei proposes slowing AI development through coordination among democracies whilst widening their lead over China. He argues that “these measures increase the leverage held by democracies and make an agreement more likely in the future.” Unfortunately, I think this assessment is overly optimistic. Those same measures could strengthen China’s incentives to accelerate its own development and resist an agreement intended to preserve its competitors’ advantage. I see similar limitations in many proposals from frontier AI lab CEOs, academics, and think tanks. Eliezer Yudkowsky and Nate Soares do a good job of explaining the likely outcome of developing advanced AI, yet they do not propose a structurally compatible solution.
I will explain this using the two foundations laid out earlier: the visibility problem surrounding AI governance and the structural constraints affecting states’ decisions.
The visibility problem surrounding AI
First, let me approach the problem from an uncommon angle.
One of the first things people notice about AI is how much it resembles science fiction. Its potential, its lack of precedent, and the speed of its development all contribute to this impression. I want to begin by examining AI’s visibility problem from a communication perspective.
The way language works, in a nutshell, is that you (the emitter) trust someone (the receiver) to have the same knowledge as you do (or close enough) of a certain word and combination of words, when you use them, therefore expecting him to receive the same message as you formulate it in your mind (or as close as possible given your ability to put it into words).
However, language is far from precise. Describing subjective things like emotions or feelings is terribly hard because of the very fact that it is subjective. The words "terrible" and "beautiful" are very imprecise because there is no clear objective way to correlate the word with a real-world "object" as that correlation requires both context and perception.
Similarly, when we are to describe things we have not seen or experienced before, we often “lack words” to describe them. Or we use the wrong words for it. This is, one, because if the "object" to describe has never happened before, then there is no precedent for describing it, and, two, because we are likely to not even understand what the object is, how it came to be or what its purpose is, or other "metadata" that would help us define it.
This is exactly what is happening now with AI. I mean, the very word "AI" is used for very different undefined things. When used to refer to AI as a "whole"—as the industry or as the sum of all models, or the top models—it assumes the receiver understands it as well as you do, which is btw a massive assumption given the novelty, scale and pace of development. Sometimes AI is used anthropomorphically, describing it as if it has drives or agency the way we understand humans do. Some other times it is used to describe capabilities or outcomes in a way that assumes both the emitter and receiver know what they are talking about, often casually jumping into the shoes of the most cutting-edge experts working on AI research. Words such as “think,” “drive,” and “feel” carry associations with humans, living things, agency, or consciousness.
Do you even realize how shocking this is to most? Casually applying concepts tied to humans, or perhaps other living things, to "machines" is a huge leap. So huge, it even creates a disconnection for the receiver. Like a massive "WTF is he even talking about?" It is not only the fact that we have never had "machines plotting before"; it is that the closest thing we have to it is basically Science Fiction, and a much more far-fetched version of it than flying cars btw. So not only do we not have precedents; it is that the closest related thoughts come entirely from imagination and fiction. Linguistically speaking, this makes effective communication using these terms very hard, to put it mildly.
The same happens with artificial general intelligence (AGI) or artificial superintelligence (ASI). We do not have a clear definition of either, as the very definition often used is vague and non-verifiable (at least not until we are very far beyond the invisible/undefined threshold). Let's take AGI as AI being more intelligent than all the experts in all fields. What does "that" even look like? What even is "it"? It is no wonder that we keep moving the bar by which we would have previously assessed "this is AGI". This is not to say that "AGI" doesn't exist; it is to say we do not know what exactly it even is or means to begin with. The concept is distant from familiar examples, both in its lack of precedent and in its possible effects.
Think of all the terms we use to describe AI and how they are hardly fitting. To say that AI "thinks" would require an accurate description of what thinking for an AI means versus how we understand we think. In fact, it is even more mind-bending than that. Because, to some degree, they seem to think. They seem to feel. At the end of the day, this is what I think of you too. It is only through repetition that I come to assume you are not a simulated NPC but, more likely, someone more closely resembling myself. So where is the threshold, if there is even one, between knowing that you think and knowing that an AI thinks? What’s going to happen when I see a robot that is physically indistinguishable from a human, at least by sight, and acts and talks like a human? So the reason these terms are mind-bending when used around AI is that they are both far (seems SciFi) and so not far (seems more real than yesterday) at the same time.
We use language to the best of our ability. The reason why we sometimes feel "wow, he's such a good speaker or writer" is because we marvel at how others have a much better ability to put into words complex topics than we do. This implies that even though we all use language, we do not use it equally well. Approximation and imprecision are everywhere. Much more so when we do not even know what we are talking about!
The same goes for predictions and percentages of likelihood. A 10% chance of human extinction. Giving it a number like that makes it feel more objective than it really is and at the same time, it comes with its own issues. Assuming we had a crystal ball that would accurately forecast chances of future outcomes, what would be the percentage of human extinction that would warrant action? Moreover, what percentage would warrant what action? Would 5% warrant waging war against states who think 5% is too low and that we should only stop AI development when reaching 15%? Understandably, that % is used to describe the fact that it can happen, but really, the problem is not merely that it can happen; the problem is emotionally reconciling ourselves to the possibility regardless of the percentage.
And one clarification. Even if we could describe "extinction", or death, or consciousness or other difficult terms, it doesn't mean we have an internal model of them; it doesn't mean we "understand" them. Take "a mother losing her son"; we can understand the linguistic meaning of the phrase and intuitively grasp the feelings associated, but we can't understand what it really is unless we have lived through it. We are driven by emotions and feelings, not by reasoning or deterministic goals. This is why stories work well for us btw, because they help build perspective and context around a concept, helping us evoke the same feelings (or close) the emitter wanted to convey much more efficiently than otherwise.
And then comes the harder part. Talking about things when we don't know what we are talking about, in order to do something about them and/or to forecast associated outcomes. I am not even joking. From AI extinction risks to loss of jobs, curing all diseases and extending life, among other wild scenarios. WTF does extinction even mean? We can't even understand what our individual personal death “means”, much less the death of the rest of humans, much less the impossibility of more humans to come after us. The same goes for all the other extreme predictions, whether good or bad. We have a very hard time reconciling ourselves to these concepts because they are so far into imagination land. I mean, when a close relative, someone who we love dearly, passes away, we tend to not believe it or to deny it. It is very hard to swallow that someone who was so dear to us would be gone. It is so hard that it takes painfully living through it to understand it. Try to explain to someone what it feels like. Then take that difficult task and multiply it by n. Or look at it from a very illustrative angle: a news headline stating "Many leading AI labs and AI safety researchers assign a substantial probability, perhaps 5–20%, to AI-driven extinction". Then the next news piece talks about politics or some other irrelevant topic. We are so utterly disconnected from the meaning of such things that we normalize them down to absurdity and ridiculousness.
This is why humility about AI’s meaning and consequences is imperative. We try to educate one another about concepts that are difficult to internalize, even for those making the arguments. AI’s visibility challenge is immense, and it calls for epistemic humility.
The structural constraints affecting AI
As with climate change, the difficulty of understanding AI’s consequences affects how states are likely to respond. Following others, I would describe AI as a hyperobject. However difficult it is to understand, states cannot afford to dismiss it. They must weigh its potential against their goals: survival and the subsidiary goals that follow from it, such as the pursuit of power.
Much has been said about this, but let me highlight the features most relevant to states’ interests.
No apparent ceiling. AI’s potential is enormous, as demonstrated by its current capabilities, the continued advance of the frontier, the capital invested, and the rapid—possibly exponential—pace of improvement. Its potential is increasingly tied to our imagination about how to use it, rather than to capabilities alone.
Extreme dual use. AI could be economically disruptive, to put it mildly, as well as militarily disruptive. Its dual-use potential extends beyond that of previous technologies.
Rapid development. The rate of development will make governments fear a narrowing window of opportunity, if it has not already done so. What if progress becomes so rapid that we cross critical thresholds before realizing that we have crossed them?
Democratization. Wider access to AI may be a double-edged sword. What if highly capable systems have their weights leaked or released and are then used by small groups for harmful purposes? This concern becomes more relevant as open-weight models become more capable and frontier models demonstrate the ability to conduct cyberattacks, discover zero-day vulnerabilities, and chain vulnerabilities to achieve a goal. Compute is not cheap, but it may become feasible for a small actor to attain a system costing, say, $5–10 million and then run enough agents using unrestricted open-weight models to disrupt critical infrastructure through a cyberattack, or cause something worse.
Potential loss of control. AI development is difficult to predict, and the systems themselves can be opaque. We train them through gradient descent over vast numbers of parameters whose interactions no human fully understands. What if we lose control? This possibility is increasingly accepted among leading experts and should be treated as a real scenario. Many of you are already familiar with it, so I will not repeat the case here.
The one-shot problem. Given the characteristics above, what if we truly have only one opportunity to get this right? What if failure at a critical step leaves no opportunity to try again? Whether through loss of control, misuse of advanced AI, or dangers we do not yet foresee, if the consequences can be so severe as to cause widespread societal disruption or even extinction, then we only get one chance. This is the one-shot problem and it resembles the black ball analogy Bostrom explained in his Vulnerable World Hypothesis paper.
All these features are subject to the visibility problem. Nonetheless, states consider, and must consider, them to varying degrees because together they present a potential threat to national security.
If these properties of AI are real, and they seem more plausible with each passing day, states will face structural pressures to act in particular ways.
Eventually, a state will be AGI-pilled enough to realize that just as they themselves are envisioning new uses and possibilities of this technology, so are their competitors. Then, the question will arise: will my competitor achieve a first-strike capability or will I, and what happens then? Nuclear deterrence is beginning to weaken due to the apparently unlimited power-creep nature of AI opening new attack vectors. Eventually, if we are not there already, we will find ourselves in a Cold War 2.0 that will grow more dangerous as AI keeps growing more capable. New capabilities, time compression, autonomous actions, etc.
As states consider these scenarios, they will feel pressure to advance because they cannot allow competitors to become stronger. If a state develops a substantially more powerful AI than its competitors and retains control over it, it will be in a position to impose its will, perhaps even achieve global hegemony or prevent others from ever catching up. This creates incentives to advance despite perceived risks. The economic and military incentives are so immense that slowing or stopping development is not feasible.
One might argue that states should negotiate agreements about what to develop and what to pause. But why have states pursued secret nuclear programs? They had incentives to evade constraints, and in some cases, such as North Korea, acquiring nuclear weapons paid off. If the incentives surrounding AI are even greater, why assume that states will not evade an agreement? A more likely scenario is that they seek ways around treaties to keep advancing AI development, driven by the security dilemma. If the United States suspects that China is secretly developing a highly capable AI, it will either seek to stop the program by whatever means are available or pursue its own program to get there first. The same applies in reverse.
Anyone who believes that mutual restraint is the likely outcome is, I’m afraid, moralistically biased and underestimates the gap between group “emergent” behavior and that of individuals. Period. The uncertainty surrounding AI, its capabilities, and its potential is so immense that states have only two credible options: permanently prevent others from developing it further, even through force, or develop it first.
Unfortunately, the current trajectory may lead to a loss of control over AI. Before that, however, we will likely see conflict between great powers, and if that happens, we may no longer have to worry about AI.
A narrow path
Is there a realistic path compatible with both the visibility problem and the structural constraints of the international system?
I want to believe there is.
Bostrom argues that, if the hypothesis (VWH paper) is true and global governance is impossible, then a possible solution to humanity’s survival is through a panopticon. I think creating such a system could itself provoke a world war, destroying what it was meant to protect and setting humanity back centuries. I do not consider this a structurally compatible solution.
Nor do I think the moral argument that “we should all agree to pause AI development because it is dangerous” will succeed. That argument assumes away both the visibility problem and the structural constraints. If we were instead confronting a massive asteroid heading toward Earth, the visibility problem would not arise, and cooperation would likely occur.
What can we do that accounts for both the visibility problem and the structural constraints of our world?
This brings us to the core issue: we need trust more than ever. Yet trust between states is elusive and often considered strategically unwise. Can we reconcile this tension sufficiently to create a narrow path? This is the task ahead.
We may not be able to stop technology’s advance, but we may be able to choose what we develop first. A highly curated sincerity assessment tool. Multimodal to maximize accuracy. Multicultural to minimize biases.
An unprecedented global effort to develop a Sincerity Assessment Tool (SAT) for the purpose of utilizing it in diplomatic settings within a concrete operationalized environment. A narrow AI model designed to assess one’s conviction in one’s own expressed beliefs whilst in conversation, in a very concrete operationalized manner.
Two leaders sitting in a room (Faraday cage), each wearing augmented reality (AR) glasses and a wristband bracelet, with high-resolution/detail-capture cameras and microphones recording the discussion. Data from the wristband, cameras and mics would feed the AI SAT model and return real-time estimations of sincerity scoring to the opponent, on his AR glasses. One would not be able to know how he is faring in the SAT; only his opponent would. At the end of the session, a second pass is performed to validate the real-time results (or invalidate them). The AI models would be run on both states’ compute; the results would be compared; differences beyond a pre-agreed percentage would indicate meddling (tampering) or at least render the results unreliable. The AI weights would be digitally signed, just like all the hardware used. The important design choice of this operationalization mode is to make deception very hard. Model or hardware tampering would be very difficult because the other state would be running its own system and results would have to align within an agreed tolerance. The conversation would be, by design, long. Say 2 hours plus. The goal is to let the AI model establish a baseline and figure out whether attempts at deception are made; the longer the conversation lasts, the more data the model acquires, the more difficult it is to sustain deception.
The goal, however, would not be to make an infallible machine. The goal would be to offer a legitimate path for honest discussion. States would never agree to give up their ability to strategically lie/deceive. This is why the operationalization of utilizing SAT would have the following requirement: plausible declinability.
All "users" eligible to use that system would have a limited number of invocations to use the tool. The number of invocations would be unknown to everyone, to themselves as well as to anyone else. The number of invocations would also be replenished; however, when or how often would be unknown too. They would only know that the number is very scarce → say 1 to 3. They would only know whether their current number of invocations left is at least 1. Therefore, if prompted by an adversary, they could decline. They could either say they don't have invocations left or simply that they are saving whatever invocations they may have for more "important/urgent matters". They should be able to refuse without implying guilt, preserving their right to strategic posturing.
However, if they were willing to pursue a candid conversation, they could leverage SAT.
As long as using SAT remains legitimately and credibly optional, states would probably consider the value of SAT being used only to signal honesty/sincerity higher than the payoffs of gaming it, and possibly failing, therefore losing the possibility of legitimately signaling sincerity in the future.
Technically speaking, this is hard but not impossible. I imagine that states could agree on building SAT as long as strict rules are kept along the way. Along the way, the concept of "trust, but verify" would be maintained. I see the model being trained using a form of federated/decentralized training. Data would be created for the purpose of this project. A very carefully thought-out data gathering/labeling/curating process would be envisioned by top AI researchers as well as other specialists (psychologists, etc.) working on this project. The plan would involve thousands of volunteers across many nations and environments in the largest data creation effort ever done. Experts sent as supervisors of the data gathering stage from competitor states would be invited to assess whether the process of the competitor adheres to the plan.
A big effort to prevent data poisoning and therefore corruption of training would be made.
AI researchers would make efforts to design the model to be compatible with fully homomorphic encryption (FHE) inference, or use whatever other strategies they envision to protect inference data and ensure that "cheating" on inference becomes very hard and very obvious.
The principle of decentralization and verification by other parties (i.e., model dual run, comparison of results, etc.) is considered and present from the very beginning.
The plausible declinability bit is enforced using a clever implementation of blockchains and zero-knowledge proofs (ZKPs), in such a way that it is resilient against state-level attacks. Perhaps multiple blockchain levels would be required to authorize token consultation/expenditure.
This could involve using openly available blockchains (chaining them) such as Ethereum as well as possibly new ones implemented for this purpose, held/maintained by states and made quantum-resistant.
I repeat, the goal is not to be unhackable but to make hacking it expensive and obvious, just enough to shift the incentives from "let's game it" to "let's use it legitimately". The moment states see more value in "let's use it legitimately" is also the moment the system becomes usable, and equally important, it also becomes feasible - building it, cooperation, etc.
The most important concept of SAT is not its ability to assess sincerity with exact precision, but rather to become a process that provides "just enough" so that states believe that any opponent invoking it has an honest intent to signal sincerity. It also needs sufficient incentives engineering so that using the tool legitimately would be preferred over gaming it. The right combination of deterring attempts at gaming it and the peace of mind that plausible declinability is maintained is what makes this idea viable, at the political level.
Technically speaking, I have no doubt it is viable. However, I'm not sure the results would be accurate/good enough, but as long as the tool and the entire framework around it can, as a whole, offer a useful system—better than otherwise—then perhaps it would be enough to build on top. AI systems have surprised us by doing things that seemed impossible only a couple of years earlier. Why assume that sufficient data and a global commitment could not produce a system like SAT?
Why the proposal is structurally compatible
This proposal is far from being a panacea, on its own. But when framed within IR theory, it actually has something else that other proposals lack. It is both feasible and enabling.
From an IR point of view, committing to this project would not require giving anything strategically important up. Quite the opposite, commitment in itself would already be a sign towards a constructive future. This is a form of what international relations theory calls costly signaling: a state would incur some cost to demonstrate its willingness to consider candid discussions in the future. It could instead dismiss the project as intrusive or a waste of resources without surprising anyone. When intentions and commitments are so difficult to verify, especially given the stakes and the anarchic international system, joining such a project would be a small sign, with some cost attached, of interest in something positive.
Even if the project failed to produce a useful system, or delivered results too late, commitment to its development would already signal something different, meaningful, and realistically possible today.
If SAT turns out to be viable, then talks about such difficult topics become possible in ways that were unthinkable. SAT would not and could not replace human diplomacy, but it could offer very much needed support - just when we possibly need it most. In essence, SAT could offer a framework, a diplomatic ritual so to speak, where states could signal - if they so desire - sincerity, even if that sincerity does not justify commitment, as a state can change its opinion at some point, or a leader may be swapped or “forced” to change his mind. The point is that intentions matter. However abstract or automatic we make states’ assessments of international matters, at the end of the day, individuals matter; their thoughts and opinions matter. Their actions matter. Consider this small thought example: do you think the US would have responded in the same way to COVID-19 if, instead of a “lab leak” (assuming it was), it had turned out to be an intentional act by China accompanied by a threat along the lines of “there is more to come if you mess with my plans for Taiwan”? The US would not have left the action unanswered; possibly it would have been considered an act of war. The consequences of COVID-19 itself would have been the same, yet what would have unfolded afterwards would likely have been very different from what has happened. Actions and consequences matter, but so do intentions and attribution.
This is precisely what this narrow path means: carving our own path through what we know works, what we know matters and what we understand to be necessary for realistic implementation.
The whole system would be designed for that. From here we could seriously talk about limiting AI research, monitoring graphics processing units (GPUs) and data centers, and the many more challenges that will follow. Because AI development is only the beginning of the many hard challenges humanity will need to face globally.
The last argument for supporting the development of SAT is that, if this trend of new tech development continues, it is only a matter of time before lie detection systems—on steroids, thanks to AI—become a reality. Possibly a much scarier reality. This argument follows the same logic as current AI labs' racing ahead: "we can do it better/safer than the competition". But, to be fair, this proposal innovates in the sense that regulation could come before the tech—for a change. The "right to lie" should be legally protected. Any "lie detection" system should therefore be deemed illegal without leveraging a robust, verifiable, voluntary commitment system, resilient to extortion and blackmailing by design.
Whilst SAT would be targeting diplomatic settings only, it would set a precedent for the legal groundwork likely needed anyway.
Arguments against the proposal
As a proponent of SAT, I am unavoidably biased myself. But if one is to seriously consider a project like this, then thinking of downsides is a necessity.
The system becomes gameable: then it should not be used. However, I am confident it can be designed and created in such a way—particularly around the operationalization—that gaming it becomes too difficult and risky. Shifting the incentives from gaming it to using it as intended becomes a design principle.
This is not going to work—sincerity cannot be assessed reliably: this argument is strong when deciding whether to use it or not. However, it does not invalidate the effort in "trying", building it and trying it out. Clear KPIs should be defined, with clear minimum thresholds; otherwise it is a no-go. But it is worth trying anyway because we may not have a better plan and because we do not lose strategic leverage and, given the stakes, it makes sense to try. For the existing evidence and its limitations, see this review of machine-learning deception detection and the National Research Council’s assessment of polygraph testing. Neither evaluates the proposed SAT protocol.
This is the precursor to a Panopticon and therefore should never be built: I think this is the strongest argument against SAT. States or third-party actors could, if the technology exists, use it for surveillance ends. Indeed, law enforcement agencies or even judicial systems could eventually make use of this. It would be bad. But this is why regulating it beforehand is important. Moreover, by showing globally the "gold standard" of decentralized plausible declinability systems, we may set an example of how such systems should be ethically implemented. Unfortunately, this argument just doesn't magically go away. I think, ultimately, we may need to pull the argument that "the alternative is worse", or that someone will eventually develop the technology anyway. A project with an explicit purpose could create an opportunity to regulate its use before implementation. However, we may not be able to peer inside AI systems, but we surely can still manage to regulate legacy systems. Operationalization of SAT or any close relative falls into that category. We can still enforce regulation.
AGI/ASI systems could use this against us: and they could create it from scratch too. In any case, SAT would exist precisely to set the conditions right in which we can cooperate at the global level to tackle AGI development in the first place, preventing a situation in which a misaligned AI could use such a system against us.
Conclusions
It seems that most of the discussion about AI safety and pausing development is poorly directed. The technical arguments are solid, and whistleblowers’ warnings about AI dangers deserve serious attention. Yet we fail to direct the conversation properly. We expect to educate audiences about topics that even the most knowledgeable experts struggle to understand. We expect to pause the development of systems into which trillions of dollars have been invested, while failing to understand the incentives driving such a potentially powerful technology forward. All this occurs in an essentially anarchic world, without a universal authority capable of enforcing decisions, actions, and consequences.
We must acknowledge that we are developing a technology that could change the world in unprecedented and perhaps unimaginable ways. We need the humility to consider that it could be humanity’s Great Filter. We should set aside our biases and moralistic wishful thinking as far as possible and think creatively about how to address it.
This essay is intended to present the problem from a different perspective and offer an idea that, however unlikely or infeasible it may seem, could encourage further thinking in this direction. At the very least, I hope it starts a conversation.
Why AI development won’t stop and what can realistically be done about it.
If war is so terrible, why do we still actively wage it? If nuclear weapons are so powerful and dangerous that we even came up with the concept of Mutually Assured Destruction (MAD) as a consequence of using them, why do we still keep stockpiles of them? If climate change is a scientifically proven fact whose consequences are not only awful, but global-reaching, why haven’t we successfully addressed it yet? If hunger could have already been eradicated, why haven’t we done so? If many of the most renowned AI scientists have warned against AI’s reckless development, why don’t we pause its development until at least we feel more certain about how to do it safely?
These questions are complex and difficult to address in the world we live in. Or in other words: if these problems were easy to manage, they would already have been solved.
In this post, I am making the claim that two fundamental challenges stand in the way of global coordination: 1) the visibility problem, concerning our ability to understand and agree on the nature and consequences of a threat; and 2) the structural problem, concerning the incentives that drive states to act even against the collective interests of individuals.
It seems to me that most of the current discussion around AI global coordination is missing out on either or both of these problems. Moreover, addressing either in isolation won't be sufficient.
Understanding these two problems helps explain why simply calling for an AI pause is unlikely to succeed—and what a more realistic path toward global coordination might look like. The result is a highly pragmatic (and speculative too, to be fair) take on why we can’t just pause AI development due to the visibility and structural issues and a proposal of the kind of solutions I think are necessary in order to even have a chance at global coordination of the sort needed.
The visibility problem
All states affected by a global challenge need to recognize it as both real and global. They also need to see themselves as sufficiently responsible for addressing it to take action. If they are expected to contribute equally, they will probably also need to perceive the challenge as similarly serious. Otherwise, they will not feel compelled to make comparable contributions.
The structural problem
States operate within an anarchic international order. There is no authority above states that can enforce laws and treaties, even when mutually interested states have signed them. Moreover, states ultimately pursue their own survival. They may care about the well-being of neighboring states, or even all states, but their primary responsibility lies within their own borders, and their survival comes first. States must also contend with substantial uncertainty about other states’ intentions. This uncertainty is a complex product of anarchy, states’ ability to wage war, their pursuit of survival, and other factors. Nonetheless, states cooperate and compete in this environment, both economically and militarily. They recognize that survival is best secured by being strong relative to their competitors. This is Realism 101. Within international relations scholarship, there is plenty of nuance and disagreement, for example about how much domestic affairs and international institutions affect states’ behavior. Without going too deeply into theory, however, anarchy, the pursuit of survival, and uncertainty create structural conditions that affect states’ decisions.
When confronted with a global issue, a state must evaluate possible outcomes through both its own perception of the problem and the structural constraints it faces. In academia, this is perhaps best captured by the security dilemma: one state’s decision to arm itself may look like preparation for aggression to its neighbor, even if its intentions are defensive. The neighbor is then motivated to arm itself in response. This creates a competitive spiral in which neither state may seek confrontation, yet both act according to the structure of the international order. The issue extends beyond military power and appears in many kinds of cooperative initiatives. Cooperation on a particular issue is always assessed through a game-theoretic lens: “What do I gain or lose relative to my competitors?” This occurs in many, if not all, areas of international relations, from climate cooperation to support for political movements, and from international aid to aggression.
The two problems together
Think of the structural problem as the rules of the game and the visibility problem as the players’ ability to see what lies ahead.
AI governance is possibly one of the most consequential global challenges we have ever faced. But before turning to it, let’s briefly consider two other examples.
Climate change
Climate change has posed an important visibility challenge. What would the consequences of inaction be? A decision-maker cannot answer only in general terms: how exactly would inaction affect their state in the years to come? Will it rain more, and by how much? What about heat waves? What consequences should they expect? What specific action would reduce a particular risk by a given amount? How much money should they allocate, and to what or whom? These are real questions that decision-makers have faced when deciding how to respond to climate change, and they are difficult to answer. Climate change illustrates the visibility problem: its effects unfold over years and decades, affect different parts of the world unequally, and involve consequences that are not clearly defined or agreed upon.
From a structural perspective, the problem becomes even muddier. Should all states contribute equally when responsibility for the problem is so unevenly distributed? If the consequences affect some states more than others, what ensures that less affected states also cooperate, or contribute as much? If reducing my state’s CO₂ emissions means slowing the development that earlier industrialized states benefited from, why should I now have to slow my own progress? How will climate cooperation affect my position relative to competitors in the years to come? Will I be better or worse off?
Those questions have structural reasons behind them and even if states see the challenge as real, they may still strategically “defect” as they see the benefits of doing so, relative to other players, as greater.
Nuclear weapons proliferation
A very different example is nuclear weapons. From a state’s decision-making perspective, nuclear weapons differ fundamentally from climate change: their visibility is crystal clear. Their capabilities and consequences were made vivid by the bombings of Hiroshima and Nagasaki and by the many nuclear tests that followed. Through this clarity, nuclear weapons became the most effective deterrent. It became clear what the consequences of attacking a nuclear-armed state would be.
This visibility also created a powerful structural incentive for other states to pursue nuclear weapons. The security dilemma associated with nuclear proliferation was particularly vivid during the Cold War. Once the United States had nuclear weapons, the Soviet Union felt threatened and pressured to develop its own. Once both had them, each felt it needed more weapons to ensure that it could retaliate with equal or greater force if attacked. This led to a cycle of nuclear armament in which the destruction of the other side could be assured, giving rise to the concept of mutually assured destruction.
The reason the nuclear standoff never escalated into war, despite its close calls, was that all parties could clearly see the consequences. Unlike climate change, nuclear proliferation was “blessed” with immense visibility for the states involved. Had the consequences been less clear, nuclear weapons would likely have been used. I have simplified the history considerably for the sake of argument, but broadly speaking, this is the main reason nuclear weapons have not been used again in war.
AI governance
In his September 2026 post We Must Pace the Frontier, Anthropic CEO Dario Amodei proposes slowing AI development through coordination among democracies whilst widening their lead over China. He argues that “these measures increase the leverage held by democracies and make an agreement more likely in the future.” Unfortunately, I think this assessment is overly optimistic. Those same measures could strengthen China’s incentives to accelerate its own development and resist an agreement intended to preserve its competitors’ advantage. I see similar limitations in many proposals from frontier AI lab CEOs, academics, and think tanks. Eliezer Yudkowsky and Nate Soares do a good job of explaining the likely outcome of developing advanced AI, yet they do not propose a structurally compatible solution.
I will explain this using the two foundations laid out earlier: the visibility problem surrounding AI governance and the structural constraints affecting states’ decisions.
The visibility problem surrounding AI
First, let me approach the problem from an uncommon angle.
One of the first things people notice about AI is how much it resembles science fiction. Its potential, its lack of precedent, and the speed of its development all contribute to this impression. I want to begin by examining AI’s visibility problem from a communication perspective.
The way language works, in a nutshell, is that you (the emitter) trust someone (the receiver) to have the same knowledge as you do (or close enough) of a certain word and combination of words, when you use them, therefore expecting him to receive the same message as you formulate it in your mind (or as close as possible given your ability to put it into words).
However, language is far from precise. Describing subjective things like emotions or feelings is terribly hard because of the very fact that it is subjective. The words "terrible" and "beautiful" are very imprecise because there is no clear objective way to correlate the word with a real-world "object" as that correlation requires both context and perception.
Similarly, when we are to describe things we have not seen or experienced before, we often “lack words” to describe them. Or we use the wrong words for it. This is, one, because if the "object" to describe has never happened before, then there is no precedent for describing it, and, two, because we are likely to not even understand what the object is, how it came to be or what its purpose is, or other "metadata" that would help us define it.
This is exactly what is happening now with AI. I mean, the very word "AI" is used for very different undefined things. When used to refer to AI as a "whole"—as the industry or as the sum of all models, or the top models—it assumes the receiver understands it as well as you do, which is btw a massive assumption given the novelty, scale and pace of development. Sometimes AI is used anthropomorphically, describing it as if it has drives or agency the way we understand humans do. Some other times it is used to describe capabilities or outcomes in a way that assumes both the emitter and receiver know what they are talking about, often casually jumping into the shoes of the most cutting-edge experts working on AI research. Words such as “think,” “drive,” and “feel” carry associations with humans, living things, agency, or consciousness.
Do you even realize how shocking this is to most? Casually applying concepts tied to humans, or perhaps other living things, to "machines" is a huge leap. So huge, it even creates a disconnection for the receiver. Like a massive "WTF is he even talking about?" It is not only the fact that we have never had "machines plotting before"; it is that the closest thing we have to it is basically Science Fiction, and a much more far-fetched version of it than flying cars btw. So not only do we not have precedents; it is that the closest related thoughts come entirely from imagination and fiction. Linguistically speaking, this makes effective communication using these terms very hard, to put it mildly.
The same happens with artificial general intelligence (AGI) or artificial superintelligence (ASI). We do not have a clear definition of either, as the very definition often used is vague and non-verifiable (at least not until we are very far beyond the invisible/undefined threshold). Let's take AGI as AI being more intelligent than all the experts in all fields. What does "that" even look like? What even is "it"? It is no wonder that we keep moving the bar by which we would have previously assessed "this is AGI". This is not to say that "AGI" doesn't exist; it is to say we do not know what exactly it even is or means to begin with. The concept is distant from familiar examples, both in its lack of precedent and in its possible effects.
Think of all the terms we use to describe AI and how they are hardly fitting. To say that AI "thinks" would require an accurate description of what thinking for an AI means versus how we understand we think. In fact, it is even more mind-bending than that. Because, to some degree, they seem to think. They seem to feel. At the end of the day, this is what I think of you too. It is only through repetition that I come to assume you are not a simulated NPC but, more likely, someone more closely resembling myself. So where is the threshold, if there is even one, between knowing that you think and knowing that an AI thinks? What’s going to happen when I see a robot that is physically indistinguishable from a human, at least by sight, and acts and talks like a human? So the reason these terms are mind-bending when used around AI is that they are both far (seems SciFi) and so not far (seems more real than yesterday) at the same time.
We use language to the best of our ability. The reason why we sometimes feel "wow, he's such a good speaker or writer" is because we marvel at how others have a much better ability to put into words complex topics than we do. This implies that even though we all use language, we do not use it equally well. Approximation and imprecision are everywhere. Much more so when we do not even know what we are talking about!
The same goes for predictions and percentages of likelihood. A 10% chance of human extinction. Giving it a number like that makes it feel more objective than it really is and at the same time, it comes with its own issues. Assuming we had a crystal ball that would accurately forecast chances of future outcomes, what would be the percentage of human extinction that would warrant action? Moreover, what percentage would warrant what action? Would 5% warrant waging war against states who think 5% is too low and that we should only stop AI development when reaching 15%? Understandably, that % is used to describe the fact that it can happen, but really, the problem is not merely that it can happen; the problem is emotionally reconciling ourselves to the possibility regardless of the percentage.
And one clarification. Even if we could describe "extinction", or death, or consciousness or other difficult terms, it doesn't mean we have an internal model of them; it doesn't mean we "understand" them. Take "a mother losing her son"; we can understand the linguistic meaning of the phrase and intuitively grasp the feelings associated, but we can't understand what it really is unless we have lived through it. We are driven by emotions and feelings, not by reasoning or deterministic goals. This is why stories work well for us btw, because they help build perspective and context around a concept, helping us evoke the same feelings (or close) the emitter wanted to convey much more efficiently than otherwise.
And then comes the harder part. Talking about things when we don't know what we are talking about, in order to do something about them and/or to forecast associated outcomes. I am not even joking. From AI extinction risks to loss of jobs, curing all diseases and extending life, among other wild scenarios. WTF does extinction even mean? We can't even understand what our individual personal death “means”, much less the death of the rest of humans, much less the impossibility of more humans to come after us. The same goes for all the other extreme predictions, whether good or bad. We have a very hard time reconciling ourselves to these concepts because they are so far into imagination land. I mean, when a close relative, someone who we love dearly, passes away, we tend to not believe it or to deny it. It is very hard to swallow that someone who was so dear to us would be gone. It is so hard that it takes painfully living through it to understand it. Try to explain to someone what it feels like. Then take that difficult task and multiply it by n. Or look at it from a very illustrative angle: a news headline stating "Many leading AI labs and AI safety researchers assign a substantial probability, perhaps 5–20%, to AI-driven extinction". Then the next news piece talks about politics or some other irrelevant topic. We are so utterly disconnected from the meaning of such things that we normalize them down to absurdity and ridiculousness.
For published survey results behind this kind of discussion, see Thousands of AI Authors on the Future of AI, which reports responses from 2,778 AI researchers.
This is why humility about AI’s meaning and consequences is imperative. We try to educate one another about concepts that are difficult to internalize, even for those making the arguments. AI’s visibility challenge is immense, and it calls for epistemic humility.
The structural constraints affecting AI
As with climate change, the difficulty of understanding AI’s consequences affects how states are likely to respond. Following others, I would describe AI as a hyperobject. However difficult it is to understand, states cannot afford to dismiss it. They must weigh its potential against their goals: survival and the subsidiary goals that follow from it, such as the pursuit of power.
Much has been said about this, but let me highlight the features most relevant to states’ interests.
All these features are subject to the visibility problem. Nonetheless, states consider, and must consider, them to varying degrees because together they present a potential threat to national security.
If these properties of AI are real, and they seem more plausible with each passing day, states will face structural pressures to act in particular ways.
Eventually, a state will be AGI-pilled enough to realize that just as they themselves are envisioning new uses and possibilities of this technology, so are their competitors. Then, the question will arise: will my competitor achieve a first-strike capability or will I, and what happens then? Nuclear deterrence is beginning to weaken due to the apparently unlimited power-creep nature of AI opening new attack vectors. Eventually, if we are not there already, we will find ourselves in a Cold War 2.0 that will grow more dangerous as AI keeps growing more capable. New capabilities, time compression, autonomous actions, etc.
As states consider these scenarios, they will feel pressure to advance because they cannot allow competitors to become stronger. If a state develops a substantially more powerful AI than its competitors and retains control over it, it will be in a position to impose its will, perhaps even achieve global hegemony or prevent others from ever catching up. This creates incentives to advance despite perceived risks. The economic and military incentives are so immense that slowing or stopping development is not feasible.
One might argue that states should negotiate agreements about what to develop and what to pause. But why have states pursued secret nuclear programs? They had incentives to evade constraints, and in some cases, such as North Korea, acquiring nuclear weapons paid off. If the incentives surrounding AI are even greater, why assume that states will not evade an agreement? A more likely scenario is that they seek ways around treaties to keep advancing AI development, driven by the security dilemma. If the United States suspects that China is secretly developing a highly capable AI, it will either seek to stop the program by whatever means are available or pursue its own program to get there first. The same applies in reverse.
Anyone who believes that mutual restraint is the likely outcome is, I’m afraid, moralistically biased and underestimates the gap between group “emergent” behavior and that of individuals. Period. The uncertainty surrounding AI, its capabilities, and its potential is so immense that states have only two credible options: permanently prevent others from developing it further, even through force, or develop it first.
Unfortunately, the current trajectory may lead to a loss of control over AI. Before that, however, we will likely see conflict between great powers, and if that happens, we may no longer have to worry about AI.
A narrow path
Is there a realistic path compatible with both the visibility problem and the structural constraints of the international system?
I want to believe there is.
Bostrom argues that, if the hypothesis (VWH paper) is true and global governance is impossible, then a possible solution to humanity’s survival is through a panopticon. I think creating such a system could itself provoke a world war, destroying what it was meant to protect and setting humanity back centuries. I do not consider this a structurally compatible solution.
Nor do I think the moral argument that “we should all agree to pause AI development because it is dangerous” will succeed. That argument assumes away both the visibility problem and the structural constraints. If we were instead confronting a massive asteroid heading toward Earth, the visibility problem would not arise, and cooperation would likely occur.
What can we do that accounts for both the visibility problem and the structural constraints of our world?
This brings us to the core issue: we need trust more than ever. Yet trust between states is elusive and often considered strategically unwise. Can we reconcile this tension sufficiently to create a narrow path? This is the task ahead.
We may not be able to stop technology’s advance, but we may be able to choose what we develop first. A highly curated sincerity assessment tool. Multimodal to maximize accuracy. Multicultural to minimize biases.
An unprecedented global effort to develop a Sincerity Assessment Tool (SAT) for the purpose of utilizing it in diplomatic settings within a concrete operationalized environment. A narrow AI model designed to assess one’s conviction in one’s own expressed beliefs whilst in conversation, in a very concrete operationalized manner.
Two leaders sitting in a room (Faraday cage), each wearing augmented reality (AR) glasses and a wristband bracelet, with high-resolution/detail-capture cameras and microphones recording the discussion. Data from the wristband, cameras and mics would feed the AI SAT model and return real-time estimations of sincerity scoring to the opponent, on his AR glasses. One would not be able to know how he is faring in the SAT; only his opponent would. At the end of the session, a second pass is performed to validate the real-time results (or invalidate them). The AI models would be run on both states’ compute; the results would be compared; differences beyond a pre-agreed percentage would indicate meddling (tampering) or at least render the results unreliable. The AI weights would be digitally signed, just like all the hardware used. The important design choice of this operationalization mode is to make deception very hard. Model or hardware tampering would be very difficult because the other state would be running its own system and results would have to align within an agreed tolerance. The conversation would be, by design, long. Say 2 hours plus. The goal is to let the AI model establish a baseline and figure out whether attempts at deception are made; the longer the conversation lasts, the more data the model acquires, the more difficult it is to sustain deception.
The goal, however, would not be to make an infallible machine. The goal would be to offer a legitimate path for honest discussion. States would never agree to give up their ability to strategically lie/deceive. This is why the operationalization of utilizing SAT would have the following requirement: plausible declinability.
All "users" eligible to use that system would have a limited number of invocations to use the tool. The number of invocations would be unknown to everyone, to themselves as well as to anyone else. The number of invocations would also be replenished; however, when or how often would be unknown too. They would only know that the number is very scarce → say 1 to 3. They would only know whether their current number of invocations left is at least 1. Therefore, if prompted by an adversary, they could decline. They could either say they don't have invocations left or simply that they are saving whatever invocations they may have for more "important/urgent matters". They should be able to refuse without implying guilt, preserving their right to strategic posturing.
However, if they were willing to pursue a candid conversation, they could leverage SAT.
As long as using SAT remains legitimately and credibly optional, states would probably consider the value of SAT being used only to signal honesty/sincerity higher than the payoffs of gaming it, and possibly failing, therefore losing the possibility of legitimately signaling sincerity in the future.
Technically speaking, this is hard but not impossible. I imagine that states could agree on building SAT as long as strict rules are kept along the way. Along the way, the concept of "trust, but verify" would be maintained. I see the model being trained using a form of federated/decentralized training. Data would be created for the purpose of this project. A very carefully thought-out data gathering/labeling/curating process would be envisioned by top AI researchers as well as other specialists (psychologists, etc.) working on this project. The plan would involve thousands of volunteers across many nations and environments in the largest data creation effort ever done. Experts sent as supervisors of the data gathering stage from competitor states would be invited to assess whether the process of the competitor adheres to the plan.
A big effort to prevent data poisoning and therefore corruption of training would be made.
AI researchers would make efforts to design the model to be compatible with fully homomorphic encryption (FHE) inference, or use whatever other strategies they envision to protect inference data and ensure that "cheating" on inference becomes very hard and very obvious.
The principle of decentralization and verification by other parties (i.e., model dual run, comparison of results, etc.) is considered and present from the very beginning.
The plausible declinability bit is enforced using a clever implementation of blockchains and zero-knowledge proofs (ZKPs), in such a way that it is resilient against state-level attacks. Perhaps multiple blockchain levels would be required to authorize token consultation/expenditure.
This could involve using openly available blockchains (chaining them) such as Ethereum as well as possibly new ones implemented for this purpose, held/maintained by states and made quantum-resistant.
I repeat, the goal is not to be unhackable but to make hacking it expensive and obvious, just enough to shift the incentives from "let's game it" to "let's use it legitimately". The moment states see more value in "let's use it legitimately" is also the moment the system becomes usable, and equally important, it also becomes feasible - building it, cooperation, etc.
The most important concept of SAT is not its ability to assess sincerity with exact precision, but rather to become a process that provides "just enough" so that states believe that any opponent invoking it has an honest intent to signal sincerity. It also needs sufficient incentives engineering so that using the tool legitimately would be preferred over gaming it. The right combination of deterring attempts at gaming it and the peace of mind that plausible declinability is maintained is what makes this idea viable, at the political level.
Technically speaking, I have no doubt it is viable. However, I'm not sure the results would be accurate/good enough, but as long as the tool and the entire framework around it can, as a whole, offer a useful system—better than otherwise—then perhaps it would be enough to build on top. AI systems have surprised us by doing things that seemed impossible only a couple of years earlier. Why assume that sufficient data and a global commitment could not produce a system like SAT?
Why the proposal is structurally compatible
This proposal is far from being a panacea, on its own. But when framed within IR theory, it actually has something else that other proposals lack. It is both feasible and enabling.
From an IR point of view, committing to this project would not require giving anything strategically important up. Quite the opposite, commitment in itself would already be a sign towards a constructive future. This is a form of what international relations theory calls costly signaling: a state would incur some cost to demonstrate its willingness to consider candid discussions in the future. It could instead dismiss the project as intrusive or a waste of resources without surprising anyone. When intentions and commitments are so difficult to verify, especially given the stakes and the anarchic international system, joining such a project would be a small sign, with some cost attached, of interest in something positive.
Even if the project failed to produce a useful system, or delivered results too late, commitment to its development would already signal something different, meaningful, and realistically possible today.
If SAT turns out to be viable, then talks about such difficult topics become possible in ways that were unthinkable. SAT would not and could not replace human diplomacy, but it could offer very much needed support - just when we possibly need it most. In essence, SAT could offer a framework, a diplomatic ritual so to speak, where states could signal - if they so desire - sincerity, even if that sincerity does not justify commitment, as a state can change its opinion at some point, or a leader may be swapped or “forced” to change his mind. The point is that intentions matter. However abstract or automatic we make states’ assessments of international matters, at the end of the day, individuals matter; their thoughts and opinions matter. Their actions matter. Consider this small thought example: do you think the US would have responded in the same way to COVID-19 if, instead of a “lab leak” (assuming it was), it had turned out to be an intentional act by China accompanied by a threat along the lines of “there is more to come if you mess with my plans for Taiwan”? The US would not have left the action unanswered; possibly it would have been considered an act of war. The consequences of COVID-19 itself would have been the same, yet what would have unfolded afterwards would likely have been very different from what has happened. Actions and consequences matter, but so do intentions and attribution.
This is precisely what this narrow path means: carving our own path through what we know works, what we know matters and what we understand to be necessary for realistic implementation.
The whole system would be designed for that. From here we could seriously talk about limiting AI research, monitoring graphics processing units (GPUs) and data centers, and the many more challenges that will follow. Because AI development is only the beginning of the many hard challenges humanity will need to face globally.
The last argument for supporting the development of SAT is that, if this trend of new tech development continues, it is only a matter of time before lie detection systems—on steroids, thanks to AI—become a reality. Possibly a much scarier reality. This argument follows the same logic as current AI labs' racing ahead: "we can do it better/safer than the competition". But, to be fair, this proposal innovates in the sense that regulation could come before the tech—for a change. The "right to lie" should be legally protected. Any "lie detection" system should therefore be deemed illegal without leveraging a robust, verifiable, voluntary commitment system, resilient to extortion and blackmailing by design.
Whilst SAT would be targeting diplomatic settings only, it would set a precedent for the legal groundwork likely needed anyway.
Arguments against the proposal
As a proponent of SAT, I am unavoidably biased myself. But if one is to seriously consider a project like this, then thinking of downsides is a necessity.
For the existing evidence and its limitations, see this review of machine-learning deception detection and the National Research Council’s assessment of polygraph testing. Neither evaluates the proposed SAT protocol.
Conclusions
It seems that most of the discussion about AI safety and pausing development is poorly directed. The technical arguments are solid, and whistleblowers’ warnings about AI dangers deserve serious attention. Yet we fail to direct the conversation properly. We expect to educate audiences about topics that even the most knowledgeable experts struggle to understand. We expect to pause the development of systems into which trillions of dollars have been invested, while failing to understand the incentives driving such a potentially powerful technology forward. All this occurs in an essentially anarchic world, without a universal authority capable of enforcing decisions, actions, and consequences.
We must acknowledge that we are developing a technology that could change the world in unprecedented and perhaps unimaginable ways. We need the humility to consider that it could be humanity’s Great Filter. We should set aside our biases and moralistic wishful thinking as far as possible and think creatively about how to address it.
This essay is intended to present the problem from a different perspective and offer an idea that, however unlikely or infeasible it may seem, could encourage further thinking in this direction. At the very least, I hope it starts a conversation.
Thank you.