I have experienced this also, though roughly a month ago, after an extended debate on trans* issues specifically.

I responded by messaging the person I had argued with, and politely asking that, if it was them who had been downvoting me, they please stop going through my comment history. I got no response, but the stream of downvotes seemed to tail off shortly thereafter.

EDIT: As a side note, the person with whom I had been debating/arguing was the same one that showed up in the thread ChrisHallquist linked. It looks like it's a pattern of behavior for him.