From Sunstein's Worst-Case Scenarios:
Our intuitions can lead to both too much and too little concern with low-probability risks. When judgments are based on analysis, they are more likely to be accurate, certainly if the analysis can be trusted. But for most people, intuitions rooted in actual experience are a much more powerful motivating force. An important task, for individuals and institutions alike, is to ensure that error-prone intuitions do not drive behavior.
More (#5) from Worst-Case Scenarios:
...Objection 4: [Knightian] uncertainty is too infrequent to be a genuine source of concern for purposes of policy and law Perhaps regulatory problems, including those mentioned here, hardly ever involve genuine uncertainty. Perhaps regulators are usually able to assign probabilities to outcomes; and where they cannot, perhaps they can instead assign probabilities to probabilities (or where this proves impossible, probabilities to probabilities of probabilities). For example, we have a lot of information about the orbits
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: