From The Visioneers:
[Gerard] O’Neill also looked to the technological changes he had seen in his own lifetime as another means of setting boundary conditions. When he was a boy, a DC-3 airliner could carry only a few dozen passengers some thousand miles. Three decades later, a Boeing 747 carried hundreds of passengers on nonstop transoceanic flights. Similarly, O’Neill considered the massive growth in computing power that he had witnessed over his twenty years as a research scientist. O’Neill extended the same sort of extrapolations to the tools of space travel. As a result, projections we might dismiss today as wildly overoptimistic appeared less so circa 1973, in the wake of a decade that saw such spectacular American and Soviet successes in space.
And:
Given its relative simplicity— the underlying physics dated to the late nineteenth century, and there were few moving parts other than the buckets— O’Neill saw the mass driver as an elegant solution. It violated no laws of physics and, after O’Neill carefully worked out its power requirements, dimensions, and performance, he concluded that it could be built with existing or soon-to-be available equipment. But, like the rest of his designs for space settlements, the feasibility of O’Neill’s mass driver was predicated on a series of optimistic assumptions and extrapolations: a lunar outpost could be established, physicists would see continued progress in improving the capabilities of superconducting wires, and so forth. And all this, of course, rested on another, broader set of assumptions about economics, the accuracy of NASA’s long-term projections, and sufficient public support. As O’Neill and other visioneers all discovered, just having a sound set of calculations, some inspiring drawings, and a vision for the future wasn’t enough.
And:
Starting in the early 1990s, a backlash against Drexler and Drexlerian nanotechnology emerged and grew. When Science published a 1991 piece called “The Apostle of Nanotechnology,” its title employed a trope frequently used in attacks on Drexler’s ideas. Journalists regularly used words like “messiah,” “guru,” “prophet,” and “nanoevangelist” to describe Drexler and, displaying a willingness to span the biblical testaments, critics likened him to both Moses and John the Baptist. Phillip Barth, an engineer at computer giant Hewlett-Packard, took this analogy even further when his posting to the Internet discussion group “sci.nanotech” speculated as to whether “nanoism” might become the “next great mass-movement” in the tradition of Christianity, Islam, or communism. When interviewed for Science, Barth dismissed Drexler’s visioneering, saying “you might as well call it nanoreligion.”
And:
scientists have often taken issue with colleagues’ popularizing activities, sometimes expressing the view that one should engage the public only at the end of one’s research career. For instance, Carl Sagan’s “vulgar” works (most notably the television series Cosmos, which he did midcareer) supposedly sabotaged his election to the National Academy of Sciences. Gerard O’Neill, meanwhile, became an advocate for space colonies and a public figure only after two decades of work as a respected physicist at an Ivy League school. Drexler, however, broke from this pattern, publishing his modest oeuvre of “real” research only after promoting nanotechnology in a popular book.
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: