To say the most obvious thing, the quality threshold for comments should be much lower than for articles. And maybe these should be also some "chat" area where comments just appear and disappear without voting, so that no one would hesitate to post there; and then after receiving some positive feedback they would feel comfortable with posting regular comments.

Maybe there could be a special posting mode for newcomers, which would provide some advantages and disadvantages, like training wheels. For example it would not display negative comment karma (karma below zero would be displayed as zero), it could encourage specific verbal feedback which would be visible only to the comment author (or perhaps require downvoters to select one of predefined explanations, such as "you were rude" or "you promoted pseudoscience"), but it would also limit the number of comments per day and per thread (to prevent spamming by people who can't take a hint). After receiving enough total karma, the newbie mode would be turned off. -- That's just a quick idea, maybe completely wrong.

Or maybe we could encourage people being nice to each other by giving positive feedback additionally to upvotes. Such as "this is nice" or "thank you for the research", which would be displayed as small icons above the comment. Generally, to add some optional flavor to the numbers, whether positive or negative.