Sure. Although I wasn't clear about this, I had in mind the common case of a non-punishing downvoter who simply disagrees with the comment (or wants to see less of its ilk) without saying why. In case punishment is the desired effect, you're right - immediate is better.