I've a few self-imposed rules on downvoting, I'm wondering how other people handle it :
I never downvote someone in a thread I'm active in (like, I never downvote an answer to one of my comments). Even if I feel the answer is a troll, because I can't trust myself to be judge and party at the same time.
I never downvote someone just because I don't like him. Not that I'm on LW since long enough to really dislike anyone for now - but still. Downvoting (or upvoting) is on the content, not on the person.
I never downvote something just because I disagree. Something has to be like a troll, contain ad hominems or obvious strawmen, or be very rude (logically or on the wording) for me to downvote.
I never downvote something I don't understand. If I don't understand, it could be that it's too confuse to be understandable (which could deserve a downvote), but also that I failed on understanding, so in doubt, I abstain.
People who go back and downvote every post or comment a Less Wrong user has ever made, please, stop doing that. It's a clever way to pull information cascades in your direction but it is clearly an abuse of the content filtering system. It's also highly dishonorable. If you truly must use such tactics then downvoting a few of your enemy's top level posts is much less evil; your enemy loses the karma and takes the hint without your severely biasing the public perception of Less Wrong's discourse.
(I just lost over 200 karma in a few minutes and that'll probably continue for awhile. This happens to me every few weeks. Edit: I mean it's been happening every few weeks for a few months for a total of only three or four. Between 400 and 700 karma lost total I think? I don't mean to overstate the problem.)