I think it is much, much more likely that the topics on which you argue with JoshuaZ happen to be ones on which your opinions, or patterns of thought, or modes of expression, are not highly regarded by other LW participants.

(FWIW, I've downvoted several of your comments, none of them in this thread so far, and I haven't the faintest recollection of whether any of them were replies to JoshuaZ. I basically cannot imagine downvoting someone for disagreeing with someone who once upon a time posted an intimidatingly funny anecdote about showing up a foolish mathematics teacher.)

You make a lot of ridiculous, badly argued comments. This and each of your linked responses to JoshuaZ were worth downvoting.

I wasn't saying that you were saying he downvoted you, I was just saying that from context it seems to me that there's an apparent reason why you might have been downvoted which merely coincided with the fact that you were responding to JoshuaZ, rather than having a causal relation to that fact.