Tetlock wrote in Superforcasters that the US intelligence establishment was likely justified to believe that it was likely that Iraq was hiding WMDs. According to Tetlock their sin was that they asserted that it's certain that Iraq had WMD.
When first reading Superforcasters I didn't quite understand the situation. After reading https://theintercept.com/2015/04/10/twelve-years-later-u-s-media-still-cant-get-iraqi-wmd-story-right/ I did.
The core problem was that Saddam lost track of some of his chemical weapons. His military didn't do perfect accounting of them and they looked the same as conventional weapons. It takes an x-ray to tell his chemical weapons apart from the normal ones.
The US intercepted communications where Saddam told his units to ensure that they had no chemical weapons that inspectors could find. Of course, that communication didn't happen in English. That communication seems to have been misinterpreted by the US intelligence community as evidence that Saddam is hiding WMDs.
Nearly nobody understood that Iraq having chemical weapons and hiding them are two different systems because you need to know where your chemical weapons happen to be to hide them. On the same token, nobody publically argues that pure incompetence might be the cause of chemical weapon usage in Syria. We want to see human agency and if a chemical weapon exploded we want to know that someone is guilty of having made the decision to use them.
In a recent facebook discussion about Iraq and the value of IARPA, a person asserted that the US intelligence community only thought Iraq had WMDs because they were subject to political pressure.
We have to get better at understanding that bad events can happen without people intending them to happen.
After understanding Iraq it's interesting to look at Syria. Maybe the chemical weapons that exploded in Syria didn't explode because Assad's troops or the opposition wanted to use chemical weapons. They might have simply exploded because some idiot did bad accounting and mislabeled a chemical weapon as being a conventional weapon.
The idea that WMDs explode by accident might be too horrible to contemplate. We have to be better at seeing incompetence as a possible explanation when we want to pin the guilt for having made an evil decision on another person.
If we can believe Tetlock, the US intelligence community didn't assign any probabilities to the interpretations. Afterwards they felt like the screwed up and funded projects like the ACE challenge that Tetlock's Good Judgement Project won via IARPA to find out how things should be done.
The problem with probabilities is that that they produce accountability. If you say X is likely to happen and X happens you can pretend that you meant that it happened with 95% probability and if X doesn't happen you can pretend that your prediction was that it was supposed to happen with 60% probability. As a result there are institutional incentives against using probability.
I don't know what the status quo happens to be within the intelligence community but when I look at the current US administration, I doubt that pushing the CIA to use probabilities more for their assessments is high on the list of priorities.
At the same time getting intelligence right in cases like this is important. That's part of the reason why IARPA spending isn't as evil as weapons research and why 80,000 hours is justified in putting IARPA-Program Manager on their job board.
Did they even have "Saddam is faking it" as a possibility?