I think you should upvote or downvote comments regardless of the karma they already have.

I'm not convinced.

• Should an unambiguously baddish comment really attract infinitely many downvotes, and an unambiguously goodish one infinitely many upvotes? -- Where "infinitely many" means "potentially a lot, with the number depending on how many people see the comment rather than on its quality".
• Suppose (perhaps more realistically than my use of the word "infinitely" above would suggest) a typical comment is voted on only a smallish number of times. If a smallish fraction of voters are crazy or stupid or evil, then with "independent voting" (on average) every comment gets a smallish amount of noise in its score, which in practice means that a few comments get scored completely wrong. Whereas with "vote towards what you think is the right score for this comment", provided most of the most recent users to see a comment are sane its score should be sane.

My own practice, for what it's worth, is somewhere intermediate between "vote according to merit, ignoring existing score" and "vote towards what seems like the score merited by the comment's quality", nearer the latter than the former.