From Priest & Arkin's Top Secret America:
“It’s the soccer ball syndrome. Something happens, and they want to rush to cover it,” said Richard H. Immerman, who, until 2009, was the assistant deputy director of national intelligence for Analytic Integrity, the office that oversees analysis for all the agencies but has little power over how individual agencies conduct their work. “I saw tremendous overlap” in what analysts worked on. “There’s no systematic and rigorous division of labor.” Even the analysts at the gigantic National Counterterrorism Center (NCTC)2—established in 2003 as the pinnacle of intelligence, the repository of the most sensitive, most difficult-to-obtain nuggets of information—got low marks from intelligence officials for not producing reports that were original, or even just better than those already written by the CIA, the FBI, the National Security Agency, or the Defense Intelligence Agency.
It’s not an academic insufficiency. When John M. Custer III was the director of intelligence at U.S. Central Command, he grew angry at how little helpful information came out of the NCTC. In 2007, he visited its director at the time, retired vice admiral John Scott Redd, to say so, loudly. “I told him,” Custer explained to me, “that after four and a half years, this organization had never produced one shred of information that helped me prosecute three wars!” Redd was not apologetic. He believed the system worked well, saying it wasn’t designed to serve commanders in the field but policy makers in Washington. That explanation sounded like a poor excuse to Custer. Mediocre information was mediocre information, no matter on whose desk it landed.
Two years later, as head of the army’s intelligence school at Fort Huachuca, Arizona, Custer still got red-faced when he recalled that day and his general frustration with Washington’s bureaucracy. “Who has the mission of reducing redundancy and ensuring everybody doesn’t gravitate to the lowest-hanging fruit?” he asked. “Who orchestrates what is produced so that everybody doesn’t produce the same thing?” The answer in Top Secret America was, dangerously, nobody.
This sort of wasteful redundancy is endemic in Top Secret America, not just in analysis but everywhere. Born of the blank check that Congress first gave national security agencies in the wake of the 9/11 attacks, Top Secret America’s wasteful duplication was cultivated by the bureaucratic instinct that bigger is always better, and by the speed at which big departments like defense allowed their subagencies to grow.
More (#2) from Top Secret America:
...The interagency group briefing slide on the status of WMD consequence management again seemed designed to minimize the appearance of any loss on the part of NorthCom, but the truth of the command’s diminished status, even in this, the one area in which it had seemed to have unambiguous leadership, showed up in a final bullet: under the new arrangements, all the response units weren’t even obligated to come to the aid of NorthCom; rather, the military services could make forces available “to the greatest extent possible.”
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: