AntiFragile makes lots of interesting points, but it's clear in some cases that Taleb is running roughshod over the truth in order to support his preferred view. I've italicized the particularly lame part:
Now, one can see a possible role for basic science, but not in the way it is intended to be. For an example of a chain of unintended uses, let us start with Phase One, the computer. The mathematical discipline of combinatorics, here basic science, derived from propositional knowledge, led to the building of computers, or so the story goes... But at first, nobody had an idea what to do with these enormous boxes full of circuits as they were cumbersome, expensive, and their applications were not too widespread, outside of database management, only good to process quantities of data. It is as if one needed to invent an application for the thrill of technology. Baby boomers will remember those mysterious punch cards. Then someone introduced the console to input with the aid of a screen monitor, using a keyboard. This led, of course, to word processing, and the computer took off because of its fitness to word processing, particularly with the microcomputer in the early 1980s. It was convenient, but not much more than that until some other unintended consequence came to be mixed into it. Now Phase Two, the Internet. It had been set up as a resilient military communication network device, developed by a research unit of the Department of Defense called DARPA and got a boost the days when Ronald Reagan was obsessed with the Soviets. It was meant to allow the United States to survive a generalized military attack. Great idea, but add the personal computer plus Internet and we get social networks, broken marriages, a rise in nerdiness, the ability for a post-Soviet person with social difficulties to find a matching spouse. All that thanks to initial U.S. tax dollars (or rather budget deficit) during Reagan’s anti-Soviet crusade.
So for now we are looking at the forward arrow and at no point, although science was at some use along the way since computer technology relies on science in most of its aspects; at no point did academic science serve in setting its direction, rather it served as a slave to chance discoveries in an opaque environment, with almost no one but college dropouts and overgrown high school students along the way. The process remained self-directed and unpredictable at every step.
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: