Yes, I agree completely that if the majority of the site isn't good about upvoting what they do want, then a few people who downvote everything (and everyone) they don't want get to exert preference-implementing power far out of proportion to their numbers, and if the preferences of those people are bad for the site, then the result is bad for the site.
And I agree that this is likely the case in reality.
But in pointing that out, you're invoking a much bigger issue than the one we started out discussing, because this isn't just a problem with downvoting all comments for a given user (aka "karmassassination").
It's a problem with downvoting all comments that support or oppose a given political platform, or all comments that support or oppose a given philosophical position, or all comments that display or fail to display a given rhetorical style, or any category of comments.
It's most obvious when the category is a user, because user's can complain of abuse and our social instinct is to defend other people from abuse we consider unjustified. (An instinct and a practice I endorse.) We don't have that instinct to defend political platforms or philosophical positions or rhetorical styles, so when users exert the same degree of power to implement their (potentially site-damaging) preferences about those things, we mostly don't notice or care, and we don't come up with catchy words for it.
In any case... regardless of the scope of the issue, the question at hand is how best to address it.
You seem to be advocating addressing this by establishing a social norm of not exerting power, and treating the few people who do as norm-violators who should cool it down and be less pushy about implementing our preferences. (At least when it comes to users... perhaps you are OK with exerting that power for other categories of comments.)
I advocate instead a social norm of exerting that power, and treating the many people who don't as norm-violators who should step it up and be less lazy about implementing our preferences.
Finding and categorizing comments by user is a lot easier than finding them by political or philosophical position. I think that's more relevant than social instincts in this case.
I think you're advocating a very time intensive approach to voting behaviour. Power would concentrate in the hands of the few who have time to plow through every relevant comment in case they come across a user or an opinion that might violate their preferences. Do you have good reasons to expect these kinds of users would protect your preferences?
If what you're advocating become...
I'm just tired of the signal pollution, and would like to be able to use karma to honestly appraise the worth of my articles and posts, without seeing 80% of my downvotes come in chunks that correspond precisely to how many posts I've made since the last massive downvote spree.
EDIT to add data points:
Spurious downvoting stopped soon after I named a particular individual (not ALL downvoting stopped, but the downvotes I got all seemed on-the-level.)
One block of potentially spurious downvoting occurred approximately one week ago, but then karma patterns returned to expected levels. I consider this block dubious, because it reasonably matches what I'd expect to see if someone noticed several of my posts together and disagreed with all of them, and did not match the usual pattern of starting with the earliest or latest post that I had made and downvoting everything (it downvoted all posts in a few threads, but not in other threads), so I'm just adding for completeness.
Spurious, indiscriminate downvoting started up again approximately half an hour ago on Sunday (12/1/2013), around noon MDT.
Edit: And now on Tuesday, 12/3/2013, at 10 AM, I'm watching my karma go down again... about 30 points so far.
Edit: And now on Saturday, 12/14/2013, at 2 PM, I'm watching my karma go down again... about 15 points so far, at a rate of about 1-2 points per second.