From Caplan's The Myth of the Rational Voter:
The history of dictatorships creates a strong impression that bad policies exist because the interests of rulers and ruled diverge. A simple solution is make the rulers and the ruled identical by giving “power to the people.” If the people decide to delegate decisions to full-time politicians, so what? Those who pay the piper— or vote to pay the piper— call the tune.
This optimistic story is, however, often at odds with the facts. Democracies frequently adopt and maintain policies harmful for most people. Protectionism is a classic example. Economists across the political spectrum have pointed out its folly for centuries, but almost every democracy restricts imports. Even when countries negotiate free trade agreements, the subtext is not, “Trade is mutually beneficial,” but, “We’ll do you the favor of buying your imports if you do us the favor of buying ours.” Admittedly, this is less appalling than the Berlin Wall, yet it is more baffling. In theory, democracy is a bulwark against socially harmful policies, but in practice it gives them a safe harbor.
More (#3) from The Myth of the Rational Voter:
One striking instance of unreasoning deference: Shortly after 9/11, polls strangely found that the nation’s citizens suddenly had more faith in their government. How often can you “trust the government in Washington to do what is right”? In 2000, only 30% of Americans said “just about always” or “most of the time.” Two weeks after 9/11, that number more than doubled to 64%. It is hard to see consumers trusting GM more after a major accident forces a recall. The public’s reaction is akin to that of religious sects who mispredict the end of the world: “We believe now more than ever.”
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: