While researching a forthcoming MIRI blog post, I came across the University of York's Safety Critical Mailing List, which hosted an interesting discussion on the use of AI in safety-critical applications in 2000. The first post in the thread, from Ken Firth, reads:

...several of [Vega's] clients seek to use varying degrees of machine intelligence - from KBS to neural nets - and have come for advice on how to implement them in safety related systems... So far we have usually resorted to the stock answer "you don't — at least not for safety critical functions", but this becomes increasingly difficult to enforce, even if your legal and moral ground is sound. Customers are increasingly pleading the need for additional functionality and for utility to have precedence over safety (!!)

The thought of having to apply formal proofs to intelligent systems leaves me cold. How do you provide satisfactory assurance for something that has the ability to change itself during a continuous learning process? I can only assume that one would resort to black box testing, with all its inherent shortcomings and uncertainties - in particular, a black-box test would only apply to the version tested, and not to subsequent evolutions...

My fear is that the longer we ignore this problem, the more likely that users will simply ignore the safety community and press on regardless (precedents from US naval combat systems and commercial operating systems??). Can anyone offer pragmatic advice to customers who are likely to use IKBS anyway? Personally I think that I prefer... [to avoid] putting unvalidatable systems in the safety-critical firing line. But for how long can we continue to achieve this?

I encountered this thread via an also-interesting technical report, Harper (2000).