I was so disappointed. I thought you were going to talk about alternatives to FAI, e.g. Oracle AI. Oh well.

I think the main problem with [the term "High assurance AGI"] is that, quite reasonably, nobody will believe that we can ever get anywhere near as much assurance in the behavior of an AGI as we can in the behavior of, say, the relatively limited AI software that controls the European Train Control System. "High assurance AGI" sounds a bit like "Totally safe all-powerful demon lord." It sounds even more wildly unimaginable to AI researchers than "safe AGI."

So? You're basically saying "Friendly AI" / "High assurance AGI" is a hard problem. Well, it is. Let's not shy away from that.

I like "High assurance AGI" because it is a less inferentially distant phrase (we know what high-assurance software is), and perhaps inclusive of other approaches than those traditionally taken in FAI. I am personally going to start using this term from now on over "Friendly AI".