I tend to downvote comments that are abusive, borderline spam (nonborderline stuff I can ban), repetitive ad nauseum with other comments the same person has made recently, or -
have three or more entangled things wrong with them (where "wrong with them" doesn't include the bare fact of a non-mainstream position, but often includes the reasoning that leads to such positions).
When it's just one or two problems, or problems that exist independently and can be addressed separately, I prefer to confront the problems in a comment, and my voting thereafter is mostly dependent on how my comment is answered, if at all. (Repetition of the content of the original comment trips the "repetitive" switch, for instance, and abuse also sometimes enters the conversation at this stage.)
When it's three problems or more, all of which relate to and intertwine with each other, I feel helpless to attack any of them, because I feel like I'd have to somehow manage all of the problems simultaneously. Sometimes I click "reply" and then start to write an answer four or five times, before giving up because any grammatically correct first sentence won't do justice to the awfulness I'm reacting to. The only way to express how very bad such a comment is, is to give the entire thing a good hard swat, which means downvoting.
This is pretty much the same standard I use (save for not being able to ban spam.) I also downvote people who persist in confusions with less than three entangled problems in the face of repeated attempts at correction, but usually not when I'm attempting to provide the corrections myself because it makes me feel rude and more likely to become frustrated.
My impression is that critiques of lesswrong mainstream positions and arguments for contrary positions are received well and achieve high karma scores when they are of very high quality. Similarly posts and comments that take lesswrong mainstream positions will still be voted down if they are of very low quality. But in between there seems to be a gulf: Moderately low quality mainstream comments will stay at 0 to -1 karma while contra-mainstream comments of (apparently) similar quality score solidly negative karma, moderately high quality mainstream comments achieve good positive karma while similar quality contra-mainstream comments stay at 0 to 2.
Do you share my impression? And if this is the case, should we try to do something about it?