Therefore I suppose that lowest karma comments of high karma contributors would mostly be trivial snarky remarks downvoted for incompatibility of sense of humor. At least this is my hypothesis - let it be tested, if we can collect such comments somehow.
In response to Jack's expression of interest I went used Wei_Dai's script to download all my comments and had a search through with some regex. By approximate count the greatest number of downvoted comments were jests of the type you mentioned, followed by comments of the form "I don't approve of the grandparent either but your specific criticism Y is wrong for this logical reason". The lowest vote that I spotted was -6 for a comment along the lines of "I fundamentally disagree with your accusations of me and do not wish to continue this conversation".
The selection here is somewhat biased in as much as I am comfortable deleting comments if for any reason a conversation is unsatisfactory to me. There are quite possibly comments or jokes that would have gone into free-fall if I did not delete them when they reached -3 in 10 seconds flat. The downvoted comments that remain I either still endorse, consider important for the conversation to make sense, haven't noticed or don't care about enough to click on. (This isn't to say that deleting a comment indicates that I do not endorse it entirely. I also have no problem with choosing my battles.)
I'm afraid Jack would be disappointed in that few of the most downvoted comments seem to be about object level subject matter. Or, if they are, it is object level conversation about something that people are... passionate about. It isn't a source of ideas I have that people most disagree with, which may be interesting to see!
For the last few months I've taken up the habit of explicitly predicting how much karma I'll get for each of my contributions on LW. I picked up the habit of doing so for Main posts back in the Visiting Fellows program, but I've found that doing it for comments is way more informative.
It forces you to build decent models of your audience and their social psychology, the game theoretic details of each particular situation, how information cascades should be expected to work, your overall memetic environment, etc. It also forces you to be reflective and to expand on your gut feeling of "people will upvote this a lot" or "people will downvote this a little bit"; it forces you to think through more specifically why you expect that, and how your contributions should be expected to shape the minds of your audience on average.
It also makes it easier to notice confusion. When one of my comments gets downvoted to -6 when I expected -3 then I know some part of my model is wrong; or, as is often the case, it will get voted back up to -3 within a few hours.
Having powerful intuitive models of social psychology is important for navigating disagreement. It helps you realize when people are agreeing or disagreeing for reasons they don't want to state explicitly, why they would find certain lines of argument more or less compelling, why they would feel justified in supporting or criticizing certain social norms, what underlying tensions they feel that cause them to respond in a certain way, etc, which is important for getting the maximum amount of evidence from your interactions. All the information in the world won't help you if you can't interpret it correctly.
Doing it well also makes you look cool. When I write from a social psychological perspective I get significantly more karma. And I can help people express things that they don't find easy to explicitly express, which is infinitely more important than karma. When you're taking into account not only people's words but the generators of people's words you get an automatic reflectivity bonus. Obviously, looking at their actual words is a prerequisite and is also an extremely important habit of sane communication.
Most importantly, gaining explicit knowledge of everyday social psychology is like explicitly understanding a huge portion of the world that you already knew. This is often a really fun experience.
There are a lot of subskills necessary to do this right, but maybe doing it wrong is also informative, if you keep trying.