Yes, I've noticed that sometimes when I argue about Certain Topics with a Particular Individual a few unrelated comments of mine get downvoted for no other apparent reason, but to a much smaller scale than some other people have complained about, so I just pretend to treat it as part of the usual noise of comments downvoted for no apparent reason.

(I'm not sure the downvote on my comment about How the Hippies Saved Physics is part of the same pattern: ChristianKl's reply was also downvoted, so I just chalk it to someone who disliked the book.)