Normally testing is done in an offline "testing" mode - using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite - but that's usually no big deal.
I don't think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM will be widely ignored. This isn't exactly the first time that people have claimed that the world is going to end.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
I declare this Open Thread open for discussion of Less Wrong topics that have not appeared in recent posts.