because it's OP's account of it and OP is third party

Yes. I think it was impolite of her to reply to something she had overheard, without knowing anything about the interactions between those two people. For all she knew, there might have been mutual knowledge among the two of them that he was in jest.

EDIT: Retracted. On reading the OP again, it seems that she already knew the woman and was reasonably sure that the man hadn't previously met her, so the above doesn't apply.