One problem I have with hesitation to downvote is that some mediocre comments are necessary. Healthy discussion should have the right ratio of good comments to mediocre comments, so that people may feel relaxed, and make simple observations, increasing rate of communication. And current downvote seems too harsh for this role. On the other hand, people who only make tedious comments shouldn't feel welcome. This is a tricky balance problem to solve with comment-to-comment voting.

I would downvote more, if we had a separate button, saying "mediocre", that would downvote the comment, say, by 0.3 points (or less, it needs calibration). The semantics of this button is basically that I acknowledge that I have read the comment, but wasn't impressed either way. From the interface standpoint, it should be a very commonly used button, so it should be very easy to use. Bringing this to a more standard setting, this is basically graded voting, --, - and ++ (not soft/hard voting as I suggested before though).

An average mediocre comment should have (a bit of) negative Karma. This way, people may think of good comments they make as currency for buying the right to post some mediocre ones. In this situation, being afraid to post any mediocre comments corresponds to excessive frugality, an error of judgment.

Also, this kind of economy calls for separation of comment Karma and article Karma, since the nature of contributions and their valuation are too different between these venues.