By way of confirmation, this has indeed happened to both myself and at least one other commenter previously (I'll leave it to them whether they want to reveal themselves). I had been waiting to see whether it would happen again to be sure, but we now seem to have pretty good evidence of bad faith.

As SoullessAutomaton notes, the karma itself is not much of an issue, but it's nonetheless rather disappointing to see this sort of behavior. It's not immediately clear whether there's much to be done about it other than public shaming, but as a possible means of preventing this happening again, I don't suppose there's any way to revoke the downvoting privileges of those who seem to be abusing the system?