Your own comments have an automatic upvote applied to them - this works like any other upvote. If you click "Vote up" after you've already voted something up, it cancels the upvote. You can (potentially) determine whether you've currently voted a comment up or down based on whether "Vote up" or "Vote down" is in boldface.

Trolls and flamebait get lots of replies too, and that shouldn't count towards karma.