It was/is the same for me and others, too - small blocks of downvotes on old comments until they reach your first one, and then periodic block downvotes on your recent comments.

I also suspected that it is done by hand at first, but now I am leaning towards it being done with a bot/script (something adapted from reddit most likely), since it happens to many users and the pattern is quite regular over a long time.