I didn't think of that, but do you think karma shouldn't depend on the order in which votes are made? Shouldn't a person who gets 20 downvotes followed by 20 upvotes have higher karma at the end than a person who had 20 upvotes followed by 20 downvotes? The first indicates improvement; the second indicates getting less interesting over time.
I am confused by how you're doing the computation, though. If half of X's votes on Y are positive and half are negative, I would expect to compute X's total contribution to Y as zero. I wouldn't keep a running sum of X's contribution to Y's karma on each thing Y has said. We can also go back and recompute the contribution to previous comments as X makes more comments. But I'd probably rather have an adaptive algorithm so that the score on individual comments reflects the situation at the moment the rating was made.
Even if we did it that way, though, this sensitivity is not a real problem. Nearly every adaptive algorithm or learning algorithm has that kind of sensitivity. It never matters in practice when there's enough data. Text compression algorithms don't have drastically different compression ratios if you swap text input blocks around.
I've gotten sufficient evidence from support that voiceofra has been doing retributive downvoting. I've banned them without prior notice because I'm not giving them more chances to downvote.
I'm thinking of something like not letting anyone give more than 5 downvotes/week for content which is more than a month old. The numbers and the time period are tentative-- this isn't my ideal rule. This is probably technically possible. However, my impression is that highly specific rules like that are an invitation to gaming the rules.
I would rather just make spiteful down-voting impossible (or maybe make it expensive) rather than trying to find out who's doing it. Admittedly, putting up barriers to downvoting for past comments doesn't solve the problem of people who down-vote everything, but at least people who downvote current material are easier to notice.
Any thoughts about technical solutions to excessive down-voting of past material?