My practice is to almost never downvote with comment. If I think something is wrong in an interesting way, then I up-vote it to increase its visibility (because its interesting) and reply with a comment.

If I think something is promising but in need of improvement I do the same.

If I downvote something, its usually not just because I disagree with it, but because I think its harmful in someway and I want to reduce its visibility. Commenting would tend to promote its visibility and be counterproductive towards that end. If someone is honestly confused as to why they have been down voted, they can explicitly ask and that will prompt me or others to comment.