How about having a limit to what proportion of another user's downvotes are allowed to come from one user? So if clarity gets downvoted by 20 people there are no limits to how many votes they can get from each of them, but if it is only Nier going on a spree against a new user he pretty soon runs into 5% or whatever the limit is, and then can't downvote that user any more.