My argument is symmetry, but the form that argument would take would be... extremely weak, once translated into words.

Roughly, however... you risk defining new norms, by treating downvotes as uniquely bad as compared to upvotes. We already have an issue where neutral karma is regarded by many as close-to-failure. It would accentuate that problem, and make upvotes worth less.