Or the downvoters are fast and early, the upvoters arrive later, which is what I've observed. I'm actually a bit worried about random downvoting of other users as well.

Ahh, those kind of downvotes. I get those patterns from time to time---not as many or fast as you are able to I'm sure since I'm a mere commenter. I remind myself to review my comments a day or two later so that some of the contempt for voter judgement can bleed away after I see the correction.