Many even use "semantics" to mean something like "pointless pedantry".

I have even been in a conversation (with some MENSAns) where the primary subject was actually the meaning of a particular word. One person tried to support his position by retorting that the other person's argument was "just semantics". Well, obviously, yes. But that's a literal description of the subject matter, not an excuse to use "Hah! Semantics!" as a general counterargument!

(Not that I endorse the conversation itself as especially useful, just that "Semantics! My side wins!" is very different to "Semantics! Let's not have this conversation".)