the person being hit can invoke desire utilitarianism as their justification

Indeed they can - and, for the sorts of arguments I tend to use this argument for, if they do, doing so will require them to accept certain of the axioms upon which desire utilitarianism is based, which can then be used in reverse against the argument I applied the Stick Test against.

It's not really meant to be used against people who've read through the Sequences (or equivalent), but as a teaching tool for those people who don't understand some of the basics of reasoning (which is still an annoyingly high percentage of the total population).