That response may be technically true (you don't acknowledge the relevance of the argument,) but I don't think it's usually appropriate, since the idea that something falling into a negative category could be irrelevant probably falls across a gap of inferential distance for your interlocutor. If they already got it, they probably wouldn't have made the argument in the first place.