Every useful code fix that I can imagine also blocks some valid voting behavior. But perhaps a more manual approach would help: if the karma of user X drops by more than Y points within time period Z, flag his account and have an admin review the sources of his karma loss, possibly taking disciplinary action towards the downvoters.

Downsides: takes a lot of coding to implement, requires effort on part of the admins, violates downvote anonymity.