There is no reasonable way to interpret this other than as having a stalker.

Suppose we find the list of users who downvoted your recent comments, and there are fifteen users on that list, each of whom is an active poster in their own right. What conclusion would you draw from that?

(It may be that, when we actually find that list, there is one account, or a handful of mostly inactive accounts, that represent almost all of the downvotes, in which case 'stalker' is a reasonable conclusion. But it's not the only way the data could turn out.)

And the solution to how not to catch false positives is to use some common sense.

Common sense is costly. The point to doing this algorithmically is that you get a query result that says "these are the twenty cases that might be karmassassination" instead of "these are the twenty thousand cases that might be karmassassination" or "these are the zero cases that might be karmassassination."

It's also not particularly wise to run this check just on people who complain- part of the point of this is to prevent karmassassins from driving users away, which hasn't happened to the people who stuck around to complain (somewhat)- and at least a few users have a habit of downvoting any comments complaining about karma loss because they don't like comments that complain about karma loss, and so they'll be extra likely to show up on that list.