More (#1) from Coal: A Human History:
One problem with the shiny, wood-burning engines proved hard to ignore: They spewed out a continuous shower of sparks and cinders wherever they went, "a storm of fiery snow," as Charles Dickens called it when he visited the United States. It was a beautiful display at night, but it had a predictable downside. Wood-burning trains commonly set nearby fields and forests ablaze; some said the trains burned more wood outside the firebox than inside.
The worst problems were on the train itself, since many early passenger cars were roofless, and all were made of wood. For example, the inaugural trip of the Mohawk Valley line in New York in 1831 (just a year after the opening of the Liverpool and Manchester line) was marred when red-hot cinders rained down upon passengers who, just moments before, had felt privileged to be experiencing this exciting new mode of travel. Those who had brought umbrellas opened them, but tossed them overboard after the first mile once their covers had burned away. According to one witness, "a general melee [then] took place among the deck-passengers, each whipping his neighbor to put out the fire. They presented a very motley appearance on arriving at the first station."
Sparks on another train reportedly consumed $6o,ooo worth of freshly minted dollar bills that were on board, singeing many passengers in the process; according to one complaint, some of the women, who wore voluminous and flammable dresses, were left "almost denuded." Over a thousand patents were granted for devices that attempted to stop these trains from igniting their surroundings, their cargo, and their passengers; but the real cure would come later in the century, when coal replaced wood as the fuel of choice. In the meantime, some of the more safety conscious railways had their passengers travel with buckets of sand in their laps to pour on each other when they caught fire.
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: