I think the point is that you shouldn't conclude "that you're confident you don't want this person in our community" just because "the argument is so moronic".

(Because there's too much noise with individual arguments to deduce a person's general competence.)

In other words, yes, downvote the comment - not the person.