I'd suggest just relying on replies and upvotes (or the lack of them) to do the soft calibration on what kind of stuff is preferred. I'd prefer the site to generally give people the benefit of the doubt and assume their opinions are worth countering with actual replies instead of an anonymous, unaddressable thumbs-down.

For outright trolling, spamming and other obvious noise that shouldn't even be on the site, the report feature is good.

Still another thing with the votes is that existing downvotes and upvotes are like a pheromone trail. It's a lot easier to downvote an already downvoted comment or upvote an already upvoted comment without much thought than it is to make the deliberation whether a new comment should have -1, 0 or +1 with no idea about what other people have already thought about it. This might lead to the voting system amplifying groupthink patterns.