In the case where the argument is over whether a non-mainstream person, group, or position should be taken seriously, that person/group/position often has more at stake. That means they're (a) more likely to become flustered by such a statement, and (b) more likely to be judged harshly for responding in kind.

For many non-mainstream issues I am associated with, (a) is false, but (b) is true. If anything, since in most every ideological issue I am extremely non mainstream, (a) is false, because I'm accustomed to much worse than a "clever quip".

I'd say instead it's when those that spend most of their time in an ideologically homogeneous subculture interact with some other ideological subculture that they will get flustered. It's a matter of acclimation to taking a hit.