I think it is much, much more likely that the topics on which you argue with JoshuaZ happen to be ones on which your opinions, or patterns of thought, or modes of expression, are not highly regarded by other LW participants.

(FWIW, I've downvoted several of your comments, none of them in this thread so far, and I haven't the faintest recollection of whether any of them were replies to JoshuaZ. I basically cannot imagine downvoting someone for disagreeing with someone who once upon a time posted an intimidatingly funny anecdote about showing up a foolish mathematics teacher.)