That depends on the comment. Some comments display so much ignorance, that they deserve to be downvoted and hidden.

Imagine a new user, who would just assert that theory of relativity is wrong, and provide their own "theory" based on some mumbo-jumbo or misunderstanding of the basic concepts of physics. That specific comment deserves to be downvoted below zero. It is not a spam, it is not offensive, so it should not be reported to moderators. It is just too stupid. Zero is for the "meh" comments, this would be below that level.

This is different from mass-downvoting all comments of other users because someone does not agree with them for political reasons.

It seems to me that many people are thinking in a direction "design a system that cannot be abused, and it will not be abused". But anything can be abused. Imagine that we would adopt a system with upvotes only, and then we would have a separate button for "report spam". Would this be safe against abuse? A malicious user could decide to mass-report all comments of their political enemies as spam. And then, what? If the spam reports are handled automatically, it would mean that new users would suddenly find themselves blocked by the system and their comments removed. (We could make the algorithm to remove the comment only if three users report it as spam; and then the abuser creates two sockpuppet accounts.) Or if the reports are not handled automatically, then some moderator must spend hours reading them and clicking "no, this is not a spam". At that moment, wouldn't it be just much simpler to ban the offender? Or perhaps remove from them specifically the ability to report spam? Analogically, we can ban the user now, or perhaps make a change that will prevent this specific user from downvoting.

At this moment, there is just one specific user abusing the system. Most of the debates about whether downvotes are bad, are started by their actions. Spending energy to redesign the whole system, which works okay for N-1 users, instead of banning the 1 disruptive user, that's a waste of everyone's time.