I do that too, but have only downvoted maybe 2 or 3 of chaosmosis's comments (he's nor particularly trollish or obnoxious, just a bit rude and obstinate; I don't know (or care) what the original disagreement on HPMoR was).

It's fairly likely that a particularly stupid or rude comment in the recent comments can trigger many people independently doing "mild karmassissanation" (checking the user's recent comments, and downvoting a couple stupid posts), giving an overall impression of systematic downvoting.