"If it's worth replying to, clearly it can't be so bad that I ought to vote it down to -4"
I'm not sure how I'd even notice if this norm were in place or not. After all, a -4 comment with replies is not evidence that there exists a person who both voted it down to -4 and replied.
I just found out that a new website feature was implemented 2 days ago. If a comment is voted to -4 or below, it and all replies and downstream comments from it will be hidden from Recent Comments, and further replies in that subthread will incur 5 karma points penalty. The hiding, but not karma penalty, applies retroactively to comments in that subthread posted before the -4 vote.
This seems to be worth a discussion post since most people are probably still voting things to below -3 without knowing the new consequences of doing so.