The proposals I have (not all of which are mutually exclusive) are :

1. Make comments within highly downvoted subthreads not appear on recent comments. Since the main problem with trolling is drowning out of recent comments, this will solve many of the issues. Moreover, it will discourage continued replies.

2. Have a separate section of the website where threads can be moved to or have a link to continue to. This section would have its own recent changes section. Moderators could move threads there or make it so that replies went to that section, and would be used for subthreads that are fairly downvoted. This has the advantage of quarantining the worst threads. This is a variation of an old system used at The Panda's Thumb which works well for that website.

3. Use the -5 penalty system but adjust either the trigger level or the penalty size. It isn't obvious that -3 and -5 are the best values for such a system if it is a good idea. The fact is that -3 isn't that negative as comment scores go, so something like -3 can be obtained without saying that much about a comment's quality. -5 and -5 or or -5 and -1 may be better values. The second would offer softer discouragement for more severe comments.

4. Use the penalty system but have karma penalties be restored if the comment it replies to is subsequently voted up more. This may be slightly more technically advanced, but this will help encourage people to speak out when they think a comment is unfairly downvoted.