I often find this distinction frustrating, in that people will sometimes jump to attacking what they believe is my relevance claim before making any kind of judgement on the truth or falsehood of my argument. And then they're wrong about the relevance claim. (if there even is one. Sometimes I'm just nitpicking.)

That sounds like it's sensible behaviour. If your argument doesn't really matter, they're probably better off not bothering to judge its truth or falsehood, instead spending their time and attention on something else. (Especially in a group setting where multiple people are making counterarguments, and it's better to engage with the real objections than to get bogged down dealing with pedants and derailments)