I think if I'd notice that I always tend to downvote some person's comments when I see them, I might end up looking at their history for comments I'm likely to also disfavor. But I wouldn't downvote their comments just because they're the one who made them; I'd judge each comment individually. I don't know if this is against community norms.