I'm starting to do this, but hitting that downvote button when there's nothing objectively false or stupid about a post is hard.

I don't find it hard, but whenever I vote a comment below zero for not adding anything, it just gets fixed back to zero by someone who probably wouldn't have voted otherwise.