It happens to me regularly - my karma drops by roughly the number of comments I've posted since the last time it happened. It's annoying, but since my overall karma still grows, it's clearly noise.

Also, it's possible the same person is downvoting everybody in this thread; I had a drop of about 10 points today. (I skipped posting any comments this weekend, so my overall comment rate has gotten kind of low lately.)

Perhaps someone is just trying to "win" at karma?