There's no hard and fast rule, but if you've downvoted someone more than 8 times in one day or read through someone's comment history and downvoted past the first page, you are doing something wrong.

It's not wrong if most of what the user writes is bad.

I tend to agree with the first statement as a rule of thumb. If you're reading and downvoting 8 postings in a day that you think are not worth reading (apparently), it seems like you're taking it upon yourself to punish that person, whereas I think it is better if we try to read what we consider to be valuable, and if we happen across some writing that seems bad, sure, critique it with a -1 if that seems worthwhile, but don't go on a jag reading all the bad (by your standards) writing you can find and downvoting it.