More (#1) from Soldiers of Reason:
The prisoner's dilemma is not as arcane or trivial as it might appear, for it addresses the conflict between individual and collective rationality. What is in a player's best interest, and how do you know you have chosen correctly? When applied to societies, the prisoner's dilemma has profound implications that could well determine whether a nation chooses a path to armament, conflict, and war, or disarmament, cooperation, and peace. Witness the case of Oppenheimer, father of the nuclear bomb and head of the general advisory committee to the Atomic Energy Commission. He recommended to Secretary of State Acheson that the United States not develop the hydrogen bomb, so as to provide "limitations on the totality of war and thus eliminating the fear and raising the hope of mankind." In other words, the United States would tell Stalin we will not build it, so you don't have to either. To this idealistic argument, Acheson, ever the wary diplomat, replied, "How can you persuade a paranoid adversary to 'disarm by example'?" The Truman administration echoed Acheson's skepticism and ultimately, in 1950, approved the development of the H-bomb.
And:
Jenkins pointed out that terrorists have a limited number of modes of attack. Their repertoire consists of six basic tactics: bombings, assassinations, armed assaults, kidnappings, barricade and hostage situations, and hijackings. More imitative than innovative, terrorists continue using a preferred mode until governments catch on and improve security measures. Thus, airplane hijackings and hostage taking were popular in the 1970s until hostage-rescue units were created and international treaties against hijackings were vigorously enforced. (Thus, New York's World Trade Center was attacked twice—in 1993 and 2001.)
By the mid-1980s, RAND analysts observed a very disturbing trend: terrorism was becoming bloodier. Whereas in 1968 the bombs of groups like the Croatian separatists were disarmed before they could injure anyone, by 1983 Hezbollah followers were ramming trucks full of explosives into U.S. Marine barracks in Lebanon, killing American servicemen by the score.28 This last incident brought attention to what would become the most worrying trend of all, suicide attacks by extremists in and from the Middle East.
According to RAND analysts, the first record of a suicide attack since ancient times occurred in May of 1972, when Japanese terrorists acting on behalf of Palestinian causes tossed a hand grenade into a group of Christian pilgrims at the airport in Lod, Israel. The attack claimed twenty-six victims but also exposed the terrorists to immediate retribution from security agents at the scene; two of the three terrorists were killed in what amounted to a suicide mission, similar to the "divine wind" or kamikaze immolations of World War II. RAND analysts believed this self-sacrifice shamed the Palestinians into similar action. If Japanese were willing to die for a foreign cause, Palestinians must demonstrate their readiness to sacrifice themselves for their own cause. The inevitable next step was the glorification of death in battle as the bloody gate to paradise.
This transformation in tactics gave terrorists unexpected results. By most accounts, after the bombing of the Beirut barracks, Reagan administration officials decided Lebanon was not worth the American funeral candles; the marines packed up and went home. American withdrawal from Lebanon and the Soviet defeat in Afghanistan, when conjoined to the rise in Muslim fundamentalism fueled by the financial support of Saudi Arabia, created a belief among terrorist groups that they had finally found a way to change the policies of Western powers.
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: