That's exactly what I do. I try to downvote comments based on how they're written (if they're rude or don't make sense, I downvote them) instead of what they're written about. (Though I may upvote comments based on agreeing with the content.)

That's exactly what I do too. (Although my downvote threshold is likely a tad more sensitive. :P)