Thanks a lot for your work on all this Xixidu, it's nice to have all these answers from a wide variety of experts. Thanks also to Demski and the others, of course.

It's very difficult to give an actual probability estimate for this question because of the way "badly done AI" pushes around the probability. (By definition, there should be some negative consequences, or it wasn't done badly enough...) However, I'll naively multiply the factors I've given, with some very rough numbers:

P(#0)P(#1)P(#2)P(#3)P(#4)

= .1 * .1 * .9 * .1 * .5 * .1

= .000045

I described a fairly narrow scenario, so we might expect significant probability mass to come from other possibilities. However, I think it's the most plausible. So, keeping in mind that it's very rough, let's say .0001.

There's an extra "* .1" in your expression, between your .9 and the .5, resulting in an extra zero, so your rough final tally should be something more like .001.