Some relevant quotes from Schlosser's Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety:
On January 23, 1961, a B-52 bomber took off from Seymour Johnson Air Force Base in Goldsboro, North Carolina, for an airborne alert... [Near] midnight... the boom operator of [a refueling] tanker noticed fuel leaking from the B-52’ s right wing. Spray from the leak soon formed a wide plume, and within two minutes about forty thousand gallons of jet fuel had poured from the wing. The command post at Seymour Johnson told the pilot, Major Walter S. Tulloch, to dump the rest of the fuel in the ocean and prepare for an emergency landing. But fuel wouldn’t drain from the tank inside the left wing, creating a weight imbalance. At half past midnight, with the flaps down and the landing gear extended, the B-52 went into an uncontrolled spin...
The B-52 was carrying two Mark 39 hydrogen bombs, each with a yield of 4 megatons. As the aircraft spun downward, centrifugal forces pulled a lanyard in the cockpit. The lanyard was attached to the bomb release mechanism. When the lanyard was pulled, the locking pins were removed from one of the bombs. The Mark 39 fell from the plane. The arming wires were yanked out, and the bomb responded as though it had been deliberately released by the crew above a target. The pulse generator activated the low-voltage thermal batteries. The drogue parachute opened, and then the main chute. The barometric switches closed. The timer ran out, activating the high-voltage thermal batteries. The bomb hit the ground, and the piezoelectric crystals inside the nose crushed. They sent a firing signal. But the weapon didn’t detonate.
Every safety mechanism had failed, except one: the ready/safe switch in the cockpit. The switch was in the SAFE position when the bomb dropped. Had the switch been set to GROUND or AIR, the X-unit would’ve charged, the detonators would’ve triggered, and a thermonuclear weapon would have exploded in a field near Faro, North Carolina...
The other Mark 39 plummeted straight down and landed in a meadow just off Big Daddy’s Road, near the Nahunta Swamp. Its parachutes had failed to open. The high explosives did not detonate, and the primary was largely undamaged...
The Air Force assured the public that the two weapons had been unarmed and that there was never any risk of a nuclear explosion. Those statements were misleading. The T-249 control box and ready/safe switch, installed in every one of SAC’s bombers, had already raised concerns at Sandia. The switch required a low-voltage signal of brief duration to operate — and that kind of signal could easily be provided by a stray wire or a short circuit, as a B-52 full of electronic equipment disintegrated midair.
A year after the North Carolina accident, a SAC ground crew removed four Mark 28 bombs from a B-47 bomber and noticed that all of the weapons were armed. But the seal on the ready/ safe switch in the cockpit was intact, and the knob hadn’t been turned to GROUND or AIR. The bombs had not been armed by the crew. A seven-month investigation by Sandia found that a tiny metal nut had come off a screw inside the plane and lodged against an unused radar-heating circuit. The nut had created a new electrical pathway, allowing current to reach an arming line— and bypass the ready/ safe switch. A similar glitch on the B-52 that crashed near Goldsboro would have caused a 4-megaton thermonuclear explosion. “It would have been bad news— in spades,” Parker F. Jones, a safety engineer at Sandia, wrote in a memo about the accident. “One simple, dynamo-technology, low-voltage switch stood between the United States and a major catastrophe!”
And:
On January 1, 1960, General Lauris Norstad, the supreme allied commander in Europe, placed all of NATO’s [USA-supplied] nuclear-capable units on a fifteen-minute alert, without consulting Congress. Every NATO air squadron was ordered to keep at least two fighter planes loaded with fuel and a nuclear weapon, parked near a runway. And thermonuclear warheads were mated to the intermediate-range Jupiter missiles in Italy and the Thor missiles in Great Britain. The new alert policy had the full support of President Eisenhower, who thought that NATO should be able to respond promptly to a Soviet attack...
Members of the Joint Committee on Atomic Energy visited fifteen NATO bases in December 1960, eager to see how America’s nuclear weapons were being deployed. The group was accompanied by Harold Agnew, ...an expert on how to design bombs, and how to handle them properly. At a NATO base in Germany, Agnew looked out at the runway and, in his own words, “nearly wet my pants.” The F-84F fighter planes on alert, each carrying a fully assembled Mark 7 bomb, were being guarded by a single American soldier. Agnew walked over and asked the young enlisted man, who carried an old-fashioned, bolt-action rifle, what he’d do if somebody jumped into one of the planes and tried to take off. Would he shoot at the pilot— or the bomb? The soldier had never been told what to do... Agnew realized there was little to prevent a German pilot from taking a plane, flying it to the Soviet Union, and dropping an atomic bomb.
The custody arrangements at the Jupiter missile sites in Italy were even more alarming. Each site had three missiles topped with a 1.4-megaton warhead— a weapon capable of igniting firestorms and flattening every brick structure within thirty square miles. All the security was provided by Italian troops. The launch authentication officer was the only American at the site. Two keys were required to launch the missiles; one was held by the American, the other by an Italian officer. The keys were often worn on a string around the neck, like a dog tag.
Congressman Chet Holifield, the chairman of the joint committee, was amazed to find three ballistic missiles, carrying thermonuclear weapons, in the custody of a single American officer with a handgun. “All [the Italians] have to do is hit him on the head with a blackjack, and they have got his key,” Holifield said, during a closed-door committee hearing after the trip. The Jupiters were located near a forest, without any protective covering, and brightly illuminated at night. They would be sitting ducks for a sniper. “There were three Jupiters setting there in the open— all pointed toward the sky,” Holifield told the committee. “Over $300 million has been spent to set up that little show and it can be knocked out with 3 rifle bullets.”
...Harold Agnew was amazed to see a group of NATO weapon handlers pull the arming wires out of a Mark 7 while unloading it from a plane. When the wires were pulled, the arming sequence began— and if the X-unit charged, a Mark 7 could be detonated by its radar, by its barometric switches, by its timer, or by falling just a few feet from a plane and landing on a runway. A stray cosmic ray could, theoretically, detonate it. The weapon seemed to invite mistakes... And a Mark 7 sometimes contained things it shouldn’t. A screwdriver was found inside one of the bombs; an Allen wrench was somehow left inside another. In both bombs, the loose tools could have caused a short circuit.
More (#4) from Command and Control:
...After taking the new job, Peurifoy made a point of reading the classified reports on every... major [nuclear] weapon accident, a lengthy catalog of fires, crashes, and explosions, of near misses and disasters narrowly averted. The fact that an accidental detonation had not yet happened, that a major city had not yet been blanketed with plutonium, offered little comfort. The probabilities remained unknown. What were the odds of a screwdriver, used to repair an alarm system, launching the warhead off a missile, the odds o
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: