(Crossposted from on Medium and my blog)

Nick Bostrom, of the Future of Humanity Institute, uses an evocative metaphor to describe the future of humanity’s technological development:

One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls — mostly white (beneficial) but also various shades of grey (moderately harmful ones and mixed blessings).

What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it. The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky.

The atom bomb, together with the long range bomber, marked the first time a small group of people could destroy dozens of cities in a matter of hours. The physicists who worked on the bomb knew that this invention had the capacity to threaten human civilization with unprecedented destruction. They built it out of fear that if they did not, an enemy state like Nazi Germany would build it first. With this development, the destructive power of humanity increased by several orders of magnitude.

History barely had time to catch its breath before many of these same physicists created a new type of nuclear bomb, the hydrogen bomb, that was hundreds of times more powerful than the atom bomb. They did it for the same reasons, out of fear that a rival state would build it first. The Soviet Union used the same justification to build their biological weapons program during the Cold War, producing large quantities of anthrax, plague, smallpox, and other biological weaponry. As far as we know, the United States did not have a comparably large program, but the fear that the US might have one was sufficient to motivate the Soviet leadership. Examples like this are not exceptions; they are the norm.

It’s clear from the history of warfare that the fear of a rival getting a technology first is sufficient to motivate the creation of purely destructive technology, including those that risk massive blowback from radiation, disease, or direct retaliation. This desire to get there first is not the only incentive to develop civilization-threatening technology, but it is the one that seems to drive people to take the most risks at a civilizational level.

Even when there is no perceived threat, the other motivations for technological innovation — profit, prestige, altruism, etc. — drive us to create new things. For most technologies this is good, and has enabled most of the progress of human civilization. The problem only arises when our technology becomes powerful enough to threaten civilization itself. While innovation is hard, it’s even more difficult to anticipate potentially dangerous innovations and prevent their creation. It’s made more difficult by the lack of personal incentive. We all know the names of famous inventors, but have you ever heard of a famous risk analyst who successfully prevented the development of a dangerous technology? I doubt it.

Still, while long term trends favor aggressive tech development, there are controls in place which slow the development of known dangerous technologies. The Non-Proliferation Treaty, the Biological Weapons Convention, and other efforts put pressure on states not to build new nuclear, chemical, or biological weapons, with variable success. Within their own borders, most countries create and enforce laws forbidding private citizens from researching or building weapons of mass destruction.

Some disincentives for dangerous technology development are cultural. The Asilomar Conference on Recombinant DNA in 1975 was an impressive effort by biologists to make sure their field did not create dangerous new kinds of organisms. A strong safety culture can lead to a reduction in accidents and an inclination towards safe exploration, though it’s not always clear how to create such a culture.

In the private sector, companies balance the benefits of “moving fast and breaking things” with the negative PR that comes from developing safety-critical tech without adequate safeguards. After one of Uber’s driverless cars killed a woman last year and the details were released, the poor safety practices of Uber were revealed. We can hope that the backlash from such accidents will incentivize a safer exploration of these technologies in the future. Other AI companies appear more inclined to invest in safety & robustness measures upfront. Both Deepmind and OpenAI, two leaders in deep learning research, have dedicated safety teams that focus on minimizing negative externalities from the technology. Whether such measures will be sufficient to curtail dangerous methods of exploration of powerful AI systems remains to be seen.

A limitation of current regulations is that they focus on technologies known to pose a risk to human life. Entirely new innovations or obscure technologies receive far less attention. If a particle accelerator might inadvertently set off a chain reaction that destroys all life, only their internal safety process and self-preservation instincts prevent them from taking risks on behalf of everyone. There is no international process for ensuring a new technology won’t end up being a black marble.

Avoiding black marbles is both a coordination problem and an uncertainty problem. In the long term, it’s not enough to get 90% of creators to refrain from exploring dangerous territory. Absent strong coordination mechanisms, future technological development suffers from the unilateralist’s curse. Even if no individual creator has ill-intent, those who believe their technologies prevent little risk will forge ahead, biasing the overall population of creators towards unsafe exploration.

While international coordination is necessary to prevent black marble scenarios, it is not sufficient. In some cases, it will not be easy to tell in advance which technologies will prove dangerous. Even if every country in the world agreed to share intelligence about black-marble technological threats and enforce international laws about their use, there is no guarantee a black marble would not be pulled out by accident.

There are no off-the-shelf solutions to international governance problems. We’re in new territory, and new social technology is required. It’s not clear how to design institutions which can incentivize rigorous risk analysis, the right kinds of caution, and quick responses to potentially dangerous developments. Nor is it clear how world powers can be brought to the negotiating table with the mandate to create the necessary frameworks. What is clear is that something new is necessary.

In 1946, physicists from the Manhattan Project brought together a group of prominent scientists and wrote a collection of essays about the implications of nuclear technology for the future. The volume was called One World or None, In the forward, Niels Bohr wrote the following, and though much has changed in the intervening decades, the message is as relevant in 2020 as it was in 1946. For context, Bohr is writing about the need for international governance to prevent nuclear war.

Such measures will, of course, demand the abolition of barriers hitherto considered necessary to safeguard national interests but now standing in the way of common security against unprecedented dangers. Certainly the handling of the precarious situation will demand the goodwill of all nations, but it must be recognized that we are dealing with what is potentially a deadly challenge to civilization itself. A better background for meeting such a situation could hardly be imagined than the earnest desire to seek a firm foundation for world security, so unanimously expressed by all those nations which only through united efforts have been able to defend elementary human rights. The extent of the contribution that agreement on this vital matter would make to the removal of obstacles to mutual confidence, and to the promotion of a harmonious relationship between nations can hardly be exaggerated.

Bohr believed that the threat posed by nuclear weapons was a game changer, and that strong international cooperation was the only solution. Without international control of the bomb, Bohr and other scientists believed that a global nuclear war was inevitable. Fortunately, this hasn’t come to pass, and while this is partially due to luck, it’s also due to the degree of coordination that has occurred between world powers. Efforts like the Non-Proliferation Treaty have been more successful than many believed possible. It’s been 75 years since Hiroshima, and only nine countries posses nuclear weapons.

The success we’ve had curtailing catastrophic threats has bought us time, but we shouldn’t mistake limited forms of cooperation like the UN Security Council for a global framework capable of addressing existential threats. Many scientists of the 1940s recognized that unprecedented forms of international coordination were necessary to address existentially-threatening technologies. 75 years later, we’ve all but lost this ambition, but the threats haven’t gone away. On the contrary, we continue to develop new technologies, and if the process continues we will find a black marble. Absent coordination, future technology will cause human extinction.

On the other hand, with better international systems of cooperation, we can anticipate and avert existential threats. Despite the fears of Cold War and many close calls, the world came through without a single nuclear weapon used against another nation. Designing international frameworks up to the tasks of coordination and global risk analysis will be difficult. It will be even more difficult to get existing leaders and stakeholders on board. It’s tempting to throw up our hands and call this kind of effort impossible, but I believe this would be a mistake. Despite the difficulty, real international coordination is still the best chance humanity has to avoid extinction.