(crossposted to EA forum here.)

Although I have not seen the argument made in any detail or in writing, I and the Future of Life Institute (FLI) have gathered the strong impression that parts of the effective altruism ecosystem are skeptical of the importance of the issue of autonomous weapons systems. This post explains why we think those interested in avoiding catastrophic and existential risk, especially risk stemming from emerging technologies, may want to have this issue higher on their list of concerns.

We will first define some terminology and do some disambiguation, as there are many classes of autonomous weapons that are often conflated; all classes have some issues of concern, but some are much more problematic than others. We then detail three basic motivations for research, advocacy, coordination, and policymaking around the issue:

1. Governance of autonomous weapon systems is a dry-run, and precedent, for governance of AGI. In the short term, AI-enabled weapons systems will share many of the technical weaknesses and shortcomings of other AI systems, but like general AI also raise safety concerns that are likely to increase rather than decrease with capability advances. The stakes are intrinsically high (literally life-or-death), and the context is an inevitably adversarial one involving states and major corporations. The sort of global coordination amongst potentially adversarial parties that will be required for governance of transformative/general AI systems will not arise from nowhere, and autonomous weapons offer an invaluable precedent and arena in which to build experience, capability, and best practices.
2. Some classes of lethal autonomous weapon systems constitute scalable weapons of mass destruction (which may also have a much lower threshold for first use or accidental escalation), and hence a nascent catastrophic risk.
3. By increasing the probability of the initiation and/or escalation of armed conflict, including catastrophic global armed conflict and/or nuclear war, autonomous weapons represent a very high expected cost that overwhelmingly offsets any gain in life from substituting autonomous weapons for humans in armed conflict.

Classes of autonomous weapons

Because many things with very different characteristics could fall under the rubric of “autonomous weapon systems” (AWSs) it is worth distinguishing and classifying them. First, let us split off cyberweapons – including AI-powered ones – as being an important but distinct issue. Likewise, we’ll set aside AI in other aspects of the military not directly related to the use of force, from strategy to target identification, where it serves to augment human action and decision-making. Rather, we focus on systems that have both (some form of) AI and physical armaments.

We now consider in turn these armaments’ target types, which we will break into categories of anti-personnel weapons, force-on-force (i.e. attacking manned enemy vehicles or structures) weaponry, and those targeting other autonomous weapon systems.

Anti-personnel AWSs can be further divided into lethal (or grossly injurious) ones versus nonlethal ones. While an interesting topic,^[1] we leave aside here non-lethal anti-personnel autonomous weapon systems, which have a somewhat distinct set of considerations.^[2]

We regard force-on-force systems designed to attack manned military vehicles and installations as relatively less intrinsically concerning. The targets of such weapons will, with considerably higher probability, be valid military targets rather than civilian ones, and insofar as they scale to mass damage, that damage will be to an adversary’s military. Of course if these weapons are highly effective, the manned targets they are designed to attack may quickly be replaced with unmanned ones.^[3]

This brings us to force-on-force systems that attack other autonomous weapons (anti-AWSs). These exist now, for example in the form of automated anti-missile systems, and are likely to grow more prevalent and sophisticated. These raise a nuanced set of considerations, as we’ll see. Some types are quite uncontroversial: no one has to our knowledge advocated for prohibiting, say, automated defenses on ships. On the other hand, very effective anti-ballistic missile systems could undermine the current nuclear equilibrium based on mutual assured destruction. And while the prospect of robots fighting robots rather than humans fighting humans is beguiling from the standpoint of avoiding the horrors of war, we’ll argue below that it is very unlikely for this to be a net positive.

This leads to a fairly complex set of considerations. FLI and other organizations have advocated for a prohibition against kinetic lethal anti-personnel weapons, with various degrees of distinction between anti-personnel and force-on-force lethal autonomous weapons, and various levels of concern and proposed regulation concerning some classes of force-on-force autonomous weapons. Motivations for this advocacy vary, but we start with one that is of particular important to FLI and to the EA/long-termist community.

Lethal autonomous weapons systems are an early test for AGI safety, arms race avoidance, value alignment, and governance

There are a surprising number of parallels between the issue of autonomous weapons and some of the most challenging parts of the AGI safety issue. These parallels include:

• In both cases, a race condition is both natural and dangerous;
• Military involvement is possible in AGI and inevitable for AWSs;
• Involvement by national governments is likely in AGI and inevitable for AWSs;
• Secrecy and information hazards are likely in both;
• Major ethical/responsibility concerns exist for both, perhaps more explicitly in AWSs;
• In both cases, unpredictability and loss of control are key issues.
• In both cases, early versions are potentially dangerous because of their incompetence; later versions are dangerous because of their competence.

The danger of arms races has long been recognized as a potentially existential threat in terms of AGI: if companies or countries worry that being second to realize a technology could be catastrophic for their corporate or national interest, then safety (and essentially all other) considerations will tend to fall to the wayside. When applied to autonomous weapons, “arms race” is literal rather than metaphorical but similar considerations apply. The general problem with arms races is that they are very easy to lose, but very difficult to win: you lose if you fail to compete, but you also lose if the competition leads to a situation dramatically increasing the risk to both parties, or to huge adverse side-effects; and this appears likely to be the case in autonomous weapons and AGI, just as it was in nuclear weapons.^[4] Unfortunately, the current international and national security context includes multiple parties fomenting a “great powers” rivalry between the US and China that is feeding an arms race narrative in AI in general, including in the military and potentially extending to AGI.

Managing to avoid an arms race in autonomous weapons – via multi-stakeholder international agreement and other means – would set a very powerful precedent for avoiding one more generally. Fortunately, there is reason to believe that this arms race is avoidable.^[5] The vast majority of AI researchers and developers are strongly against an arms race in AWSs,^[6] and AWSs enjoy very little popular support.^[7] Thus prohibition or strong governance of lethal autonomous weapons is a test instance in which the overwhelming majority of AI researchers and developers agree. This presents an opportunity to draw at least some line, in a globally coordinated way, between what is and is not acceptable in delegating decisions, actions, and responsibility to AI. And doing so would set a precedent for avoiding a race by recognizing that even each participant’s interests are better served by at least some coordination and cooperation.

Good governance of AWSs will take exactly the sort of multilateral cooperation, including getting militaries onboard, that is likely to be necessary with an overall AI/AGI (figurative) arms race. The methods, institutions, and ideas necessary to govern AGI in a beneficial and stable multilateral system is very unlikely to arise quickly or from nowhere. It might arise steadily from growth of current AI governance institutions such as the OECD, international standards bodies, regulatory frameworks such as that developing in the EU, etc. But these institutions tend to explicitly and deliberately exclude discussion of military issues so as to make reaching agreements easier. But this then avoids precisely the sorts of issues of national interest and military and geopolitical power that would be at the forefront of the most disastrous type of AGI race. Seeking to govern deeply unpopular AWSs (which also presently lack strong interest groups pushing for them) provides the easiest possible opportunity for a “win” in coordination amongst military powers.

Beyond race vs. cooperative dynamics, autonomous weapons and AGI present other important parallels at the level of technical AI safety and alignment, and multi-agent dynamics.

Lethal autonomous weapon systems are a special case of a more general problem in AI safety and ethics that the technical capability of being effective may be much simpler than what is necessary to be moral or ethical or legal. Indeed the gap between making an autonomous weapon that is effective (successfully kills enemies) and one that is moral (in the sense of, at minimum, being able to act in accord with international law) may be larger than in any other AI application: the stakes are so high, and the situations so complex, that the problem may well be AGI-complete.^[8]

In the short-term, then, there are complex moral questions. In particular, who is responsible for the decisions made by an AI system when the moral responsibility cannot lie with the system? If an AI system is programmed to obey the “law of war,” but then fails, who is at fault? On the flip side, what happens if the AI system “disagrees” with a human commander directing an illegal act? Even a weapon that is very effective at obeying such rules is unlikely to (be programmed to be able to) disobey a direct “order” from its user: if such “insubordination” is possible it raises the risk of incorrigible and intransigent intelligent weapons systems; but if not, it removes an existing barrier to unconscionable military acts. While these concerns are not foremost from the perspective of overall expected utility, for these and other reasons we believe that delegating the decision to take a human life to machine systems is a deep moral error, and doing so in the military sets a terrible precedent.

Things get even more complex when multiple cooperative and adversarial systems are involved. As argued below, the unpredictability and adaptability of AWSs is an issue that will increase with better AI rather than decrease. And when many such agents interact, emergent effects are likely that are even less predictable in advance. This corresponds closely to the control problem in AI in general, and indicates a quite pernicious problem of AI systems leaving humans unable to predict what they will do, or effectively intervene if what they do runs counter to the wishes of human overseers.

In advanced AI in general, one of the most dangerous dynamics is the unwarranted belief of AI developers, users, and funders that AI will – like most engineered technologies – by default do what we want it to do. It is important that those who would research, commission, or deploy autonomous weapons be fully cognizant of this issue; and we might hope that the cautious mindset this could engender could bleed into or be transplanted into safety considerations for powerful AI systems in and out of the military.

Lethal autonomous weapons as WMDs

There is a very strong case to classify some anti-personnel AWSs as weapons of mass destruction. We regard as the key defining characteristic of WMDs^[9] that a single person’s agency directed through the weapon can directly cause many fatalities with very little additional support structure (like an army to command.) This is not possible with “conventional” weapons systems like guns, aircraft, and tanks, where the deaths caused scale roughly linearly with the number of people involved in causing those deaths.

With this definition, some anti-personnel lethal AWSs (such as microdrone munition-carrying “slaughterbots”) would easily qualify. These weapons are essentially (microdrone)+(bullet)+(smartphone components), and with near-future technology and efficiency of scale, slaughterbots could plausibly be as inexpensive as $100 each to manufacture en masse. Even with a 50% success rate and doubling of the cost to account for delivery, this is $400/fatality. Nuclear weapons cost billions to develop, then tens to hundreds of millions per warhead. A nuclear strike against a major city is likely to have hundreds of thousands of fatalities (for example a 100 kiloton strike against downtown San Francisco would cause an estimated 200K fatalities and 400K injuries.) 100,000 kills worth of slaughterbots, at a cost of $40M, would be just as cost-effective to manufacture and deploy, and dramatically cheaper to develop. They are more bulky than a nuclear warhead but could plausibly still fit in a 40’ shipping container (and unlike nuclear, chemical and biological weapons are safe to transport, hard to detect, and can easily be deployed remotely.)

This is possible with near-future technology.^[10] It is not hard to imagine even more miniaturized weaponry, in a continuum that could reach all the way to nanotechnology. And unlike (to first approximation) for nuclear weapons, effectiveness and cost-efficiency are likely to significantly increase with technological improvement.^[11] Thus if even a fraction of the resources that have been put into nuclear weapons were put into anti-personnel lethal AWSs, they could potentially become as large of a threat. Consider that it took less than 20 years from the 1945 Trinity test until the Cuban Missile Crisis that almost led to a global catastrophe, and that a determined but relatively minor program by a major military could likely develop a slaughterbot-type WMD within a handful of years.

One crucial difference between AWs and other WMDs is that the former’s ability to discriminate among potential targets is much better, and this capability should increase with time. A second is that Autonomous WMD would, unlike other WMDs, leave the targeted territory relatively undamaged and quickly inhabitable.

In certain ways these are major advantages: a (somewhat more) responsible actor could use this capability to target only military personnel insofar as they are distinguishable, or target only the leadership structure of some rogue organization, without harming civilians or other bystanders. Even if such distinctions are difficult, such weapons could relatively easily never target children, the wounded, etc. And a military victory would not necessarily be accompanied by the physical destruction of an adversary’s infrastructure and economy.

The unfortunate flip-side of these differences, however, is that anti-personnel lethal AWSs are much more likely to be used. In terms of “bad actors,” along with the advantages of being safe to transport and hard to detect, the ability to selectively attack particular types of people who have been identified as worthy of killing will help assuage the moral qualms that might otherwise discourage mass killing. Particular ethnic groups, languages, uniforms, clothing, or individual identities (culled from the internet and matched using facial recognition) could all provide a basis for targeting and rationalization. And scalable destruction of physical assets would make autonomous WMDs far more strategically effective for seizing territory.

Autonomous WMDs would pose all of the same sorts of threats that other ones do,^[12] from acts of terror to geopolitical destabilization to catastrophic conflict between major powers. Tens of billions of USD are spent by the US and other states to prevent terrorist actions using WMDs and to prevent the “wrong” states from acquiring them. And recall that a primary (claimed) reason for the Iraq war (at trillions of USD in total cost) was its (claimed) possession of WMDs. It thus seems foolish in the extreme to allow – let alone implicitly encourage – the development of a new class of WMDs that could proliferate much more easily than nuclear weapons.

Lethal autonomous weapons as destabilizing elements in and out of war

On the list of most important things in the world, retaining global international peace and stability rates very highly; instability is a critical risk factor for global catastrophic or X-risk. Even nuclear weapons, probably the greatest current catastrophic risk, are arguably stabilizing against large-scale war. In contrast, there are many compelling reasons to see autonomous weapons as a destabilizing effect, perhaps profoundly so.^[13]

For a start, AWSs like slaughterbots are ideal tools of assassination and terror, hence deeply politically destabilizing. The usual obstacles to one individual killing another – technical difficulty, fear of being caught, physical risk during execution, and innate moral aversion – are all lowered or eliminated using a programmable autonomous weapon. All else being equal, if lethal AWSs proliferate, this will make both political assassinations and acts of terror inevitably more possible, and dramatically so if the current rate is limited by any of the above obstacles. Our sociopolitical systems react very strongly to both types of violence, and the consequences are unpredictable but could be very large-scale. Tallying up the economic cost of the largest terror attacks to date – those on 9/11 – surely reaches into trillions of $USD, with an accompanying social cost of surveillance, global conflict, and so on.

Second, like drone warfare, lethals AWSs are likely to further (and more widely) lower the threshold of state violence toward other states. The US, for one, has shown little reluctance to strike targets of interest in certain other countries, and lethal AWSs could diminish the reluctance even more by lowering the level of collateral damage.^[14] This type of action might spread to other countries that currently lack the US’s technical ability to accomplish such strikes. Lethal (or nonlethal) AWSs could also increase states’ ability to perpetrate violence against its own citizens; whether this increases or decreases stability of those states, seems, however, unclear.

Third, AWSs of all types threaten to upset the status quo of military power. The advantage of major military powers rests on decades of technological advantage coupled with vast levels of spending on training and equipment. A significant part of this investment and advantage would be nullified by a new class of weapon that evolves on software rather than hardware timescale. Moreover, even if the current capability “ranking” of military powers were preserved, for a weapon that strongly favors offense (as some have argued for antipersonnel AWSs) there may be no plausible technical advantage that suffices^[15] – indeed this is a key reason that major military powers are so concerned about nuclear proliferation.

Finally, and probably most worrisome, if there is an open arms race in AWSs of all types, we see a dramatically increased risk of accidental triggering or escalation of armed conflict.^[16] A crucial desirable feature of AWSs from the military point of view is to be able to understand and predict^[17] how they will operate in a given situation: under what conditions will they take action on what sorts of targets, and how. This is a very difficult technical problem because, given the variety of situations in which an AWS might be placed, it could easily fall outside the context of its training data. But it is a very crucial one: without such an understanding, fielding an AWS would raise a spectrum of potential unintended consequences.

But now consider a situation in which AWSs are designed to attack and defend against other AWSs. In this case, predictability of how a given AWS will function turns from a desirable feature (for military decision makers to understand how their weapon will function) into an exploitable liability.^[18] There will then be a very strong conflict between the desire to make an AWS predictable to its user, and the necessity of making it unpredictable and unexploitable to its adversary. This is likely to manifest as a parallel conflict between a simple set of clear and followable rules (making the AWS more predictable) versus a high degree of flexibility and “improvisation” (making the AWS more effective but less predictable.) This competition would happen alongside a competition in the speed of the OODA (Observe, Orient, Decide, Act) loop. The net effect seems to inevitably point to a situation in which AWSs react to each other in a way that is both unpredictable in advance, and too fast for humans to intervene. There seems little opportunity for such conflict between such weapons to de-escalate. Inadvertent military conflict is already a major problem when humans are involved who fully understand the stakes. It seems very dangerous to have a situation in which the ability to resist or forestall such escalation would be seen as a major and exploitable military disadvantage.

Keeping the threshold for war high is very obviously very important but it is worth looking at the numbers. A large-scale nuclear war is unbelievably costly: it would most likely kill 1-7Bn in the first year and wipe out a large fraction of Earth’s economic activity (i.e. of order one quadrillion USD or more, a decade worth of world GDP.)Some current estimates of the likelihood of global-power nuclear war over the next few decades range from ~0.5-20%. So just a 10% increase in this probability, due to an increase in the probability of conflict that leads to nuclear war, costs in expectation ~500K - 150m lives and ~$0.1-10Tn (not counting huge downstream life-loss and economic losses). Insofar as saving the lives of soldiers is an argument in favor of deploying AWSs, it seems exceedingly unlikely that substituting lethal AWSs for soldiers will ever save this many lives or value: AWSs are unlikely to save any lives in a global thermonuclear war, and it is hard to imagine a conventional war of large enough scale that AWSs could substitute for this many humans, without the war escalating into a nuclear one. In other words, imagine a war with human combatants in which $N$ are expected to die, with probability $P_n$ of that or another related war escalating into a nuclear exchange costing $M$ lives. We suppose that we might replace these $N$ human combatants with autonomous ones but at the cost of increasing the probability to $P_y$. The expected deaths are $N+p_{n}M$ in the human-combatant case and $P_{y}M$ in the autonomous combatant case, with a difference in fatalities of ($P_y-P_n\left)\right(M-N)$. Given how much larger $M$ (~1-7 Bn) is than $N$ (tens of thousands at most) it only takes a small difference $(P_y-P_n)$ for this to be a very poor exchange.

What should be done?

We’ve argued above that the issue of autonomous weapons is not simply concerns about soulless robots killing people or discomfort with the inevitable applications of AI to military purposes. Rather, particular properties of autonomous weapons seem likely to lead, in expectation, to a substantially more dangerous world. Moreover, actions to mitigate this danger may even help – via precedent and capability-building – in mitigating others. The issue is also relatively tractable – at least for now, and in comparison to more intractable-but-important issues like nuclear accident risk or the problematic business model of certain big tech companies. Although involvement of militaries makes it difficult, there is as yet relatively little strong corporate interest in the issue.^[19] International negotiations exist and are underway (though struggling to make significant headway.) It is also relatively neglected, with a small number of NGOs working at high activity, and relatively little public awareness of the issue. It is thus a good target for action by usual criteria.

Arguments against being concerned with autonomous weapons appear to fall into three general classes:^[20]The first is that autonomous weapons are a net good. The second is that autonomous weapons are an inevitability, and there’s little or nothing to be done about it. The third is simply that this is “somebody else’s problem,” and low-impact relative to other issues to which effort and resources could be devoted.^[21] We’ve argued above against all three positions: the expected utility of widespread autonomous weapons is likely to be highly negative (due to increase probability of large-scale war, if nothing else), the issue is addressable (with multiple examples of past successful arms-control agreements), currently tractable if difficult, and success would also improve the probability of positive results in even more high-stakes arenas including global AGI governance.

If the issue of autonomous weapons is important, tractable and neglected, it is worth asking what success would look like. Many of the above concerns could be substantially mitigated via an international agreement governing autonomous weapons; unfortunately they are unlikely to be significantly impacted by lesser measures. Arguments against such an agreement tend to focus on how hard or effective it would be, or conflate very distinct considerations or weapons classes. But there are many possible provisions such an agreement could include that would be net-good and that we believe many countries (including major military powers) might agree on. For example:

• Some particular, well-defined, classes of weapons could be prohibited (as biological weapons, laser blinding weapons, space-based nuclear weapons, etc., are currently). Weapons with high potential for abuse and relatively little real military advantage to major powers (like slaughterbots) should be first in line. Automated primarily defensive weaponry targeting missiles or other unmanned objects, or non-injurious AWSs, very probably should not be prohibited in general. The grey area in the middle should be worked out in multilateral negotiation.
• For whatever is not prohibited, there could be agreements (supplemented by internal regulations) regarding proliferation, tracking, attribution, human control, etc., to AWSs; for some examples see this “Roadmapping exercise,” which emerged as a sketch of consensus recommendations from a meeting between technical experts with a very wide range of views on autonomous weapons.

Highlighting the risks of autonomous weapons may also encourage militaries to invest substantially in effective defensive technologies (especially those that are non-AI and/or that are purely defensive rather than force-on-force) against lethal autonomous weapons, including the prohibited varieties. This could lead to (an imperfect but far less problematic than our current trajectory) scenario in which anti-personnel AWSs are generally prohibited, yet defended against, and other AWSs are either prohibited or governed by a strong set of agreements aimed at maintaining a stable detente in terms of AI weapons.

FLI has advanced the view – widely shared in the AI research community – that the world will be very ill-served by an arms race and unfettered buildup in autonomous weaponry. Our confidence in this is quite high. We have further argued here that the stakes are significantly greater than many have appreciated, which has motivated both FLI’s advocacy in this area as well as this posting. Less clear is how much and what can be done about the dynamics driving us in that direction. We welcome feedback both regarding the arguments put forth in this piece, and more generally about what actions can be taken to best mitigate the long-term risks that autonomous weapons may pose.

I thank FLI staff and especially Jared Brown and Emilia Javorsky for helpful feedback and notes on this piece.

Notes

---

1. As a major advantage, nonlethal autonomous weapons need not defend themselves and so can take on significant harm in order to prevent harm while subduing a human. On the other hand if such weapons become _too _effective they may make it too easy and “low-cost” for authoritarian governments to subdue their populace. ↩︎

2. Though we would note that converting a nonlethal autonomous weapon into a lethal one could require relatively small modification, as it would really amount to using the same software on different hardware (weapons). ↩︎

3. Autonomy also created new capabilities – like swarms – that are wholly new and will subvert existing weapons categories. The versatility of small, scalable, lethal AWS is of note here, as they might be quickly repurposed for a variety of target types, with many combining to attack a larger target. ↩︎

4. The claim here is not that nuclear weapons are without benefit (as they arguably have been a stabilizing influence so far), but the arms race to weapons numbers far beyond deterrence probably is. Understanding of nuclear winter laid bare the lose-lose nature of the nuclear arms race: even if one power were able to perform a magically effective first strike to eliminate all of the enemy’s weapons, that power would still find itself with a starving population. ↩︎

5. AI will be unavoidably tied to military capability, as it has appropriate roles in the military that would be unpreventable even if this were desirable. However this is very different from an unchecked arms race, and de-linking AI and weaponry as much as possible seems a net win. ↩︎

6. For example in polling for the Asilomar Principles among many of the world’s foremost AI researchers, Principle 18, “An arms race in lethal autonomous weapons should be avoided,” polled the very highest. ↩︎

7. This survey shows about 61% pro and 22% con for their use. This article points to a more recent EU poll with high (73%) support for an international treaty prohibiting them. It should be noted that both surveys were commissioned by the Campaign to Stop Killer Robot. This study argues that opinions can easily change due to additional factors, and in general we should assume that public understanding of autonomous weapons and their implications is fairly low. ↩︎

8. There is significant literature and debate on the difficulty of satisfying requirements of international law in distinction, proportionality; see e.g. this general analysis, and this discussion of general issues of human vs. machine control. Beyond the question of legality are moral questions, as explored in detail here for example. ↩︎

9. The term “WMD” is somewhat poorly defined, sometimes conflated with the trio of chemical, biological and nuclear weapons. But if we define WMDs in terms of characteristics such that the term could at least in principle apply both to and beyond nuclear, chemical and biological weapons, then it’s hard to avoid including anti-personnel AWSs. One might include additional or alternative characteristics that (a) WMDs must be very destructive, and/or (b) that they are highly indiscriminate, and/or (c) that they somehow offend human sensibilities through their mode of killing. However, (a) chemical and biological weapons are not necessarily destructive (other than to life/humans); (b) if biological weapons are made more discriminate, e.g. to attack only people with some given set of genetic markers, they would almost certainly still be classed as WMDs and arguably be of even more concern; (c) “offending sensibilities” is rather murkily defined. ↩︎

10. It has been argued that increasing levels of autonomy in loitering munition systems represent a slippery slope, behaving functionally as lethal autonomous weapons on the battlefield. Some of the systems identified as of highest concern and also of lower cost relative to large drones have been deployed in recent drone conflicts in Libya and Nagorno-Karabakh. ↩︎

11. While speculating on particular technologies is probably not worthwhile, note that the physical limits are quite lax. For example, ten million gnat-sized drones carrying a poison (or noncontagious bioweapon) payload could fit into a suitcase and fly at 1 km/hr (as gnats do). ↩︎

12. Note that autonomous WMDs could also be combined with or enable other ones: miniature autonomous weapons could efficiently deliver a tiny chemical, biological or radiological payload, combining the high lethality of existing WMDs with the precision of autonomous ones. ↩︎

13. For some analyses of this issue see this UNIDIR report and this piece. Even the dissertation by Paul Scharre, concludes that “The widespread deployment of fully autonomous weapons is therefore likely to undermine stability because of the risk of unintended lethal engagements.” and recommends regulatory approaches to mitigate the issue. ↩︎

14. In the case of the US, this effect is likely to be present even if lethal AWSs were prohibited – human-piloted microdrones or swarms should be able to provide most of the advantages as lethal AWSs, except in rare circumstances when the signal can be blocked. ↩︎

15. Israel presents a particularly important case. While its small population motivates replacing or augmenting human soldiers with machines, to us it seems unwise to seek unchecked global development of lethal AWSs, when it is surrounded by adversaries perfectly capable of developing and fielding them. ↩︎

16. This RAND publication lays out the argument in some detail. ↩︎

17. For detailed discussion of these terms, see e.g. this UNIDIR report. ↩︎

18. Autonomous weapons developers are already thinking along these lines of course; see for example this article about planning to undermine drone swarms by predicting and intervening in their dynamics. ↩︎

19. While arms manufacturers will tend to disfavor limitations on arms, few if any are currently profiting from the sorts of weapons that might be prohibited by international agreement, and there is plenty of scope for profit-making in designing defenses against lethal autonomous weapons, etc. ↩︎

20. We leave out disingenuous arguments against straw men such as “But if we give up lethal autonomous weapons and allow others to develop them, we lose the war.” No one serious, to our knowledge, is advocating this – the whole point of multilateral arms control agreements is that all parties are subject to them. Ironically, though, this self-defeating position is the one taken at least formally by the US (among others), for which current policy largely disallows (though see this re-interpretation) fully autonomous lethal weapons, even while the US argues against a treaty creating such a prohibition for other countries. ↩︎

21. A more pernicious argument that we have heard is that advocacy regarding autonomous weapons is antagonistic to the US military and government, which could lead to lack of influence in other matters. This seems terribly misguided to us. We strongly believe US national security is served, rather than hindered, by agreements and limitations on autonomous weapons and their proliferation. There is a real danger that US policymakers and military planners are failing to realize this precisely due to lack of input from experts who understand the issues surrounding AI systems best. Moreover, neither the US government nor the US military establishment are monolithic institutions, but huge complexes with many distinct agents and interests. ↩︎