The basic thrust of my argument was that it wasn't something he could just decide an answer to

If the question was whether prime factorisation was likely to become easy, then probably you'd be justified in saying, essentially, "you don't get to have an opinion!". But since RSA is only an implementation, not a pure essence of mathematics, it might be vulnerable in ways we don't know about yet. It wouldn't be the first time. (Of course your interlocutor might not have intended this interpretation.)

I think this is a good example of a common case, where our certainty concerning ideal objects like mathematics can blind us to the existence of uncertainty in the real world. If someone designs an AI tomorrow and provides a proof of its friendliness, should we implement it?

This question nicely straddles the dichotomy between taxation (which has policy consequences and is somewhat subjective) and modular arithmetic (which has no policy consequences, and "can look after itself").