I've contributed to threads where my discussion partner's every comment was downvoted, but it wasn't me. The damage isn't done just to the one being downvoted, it's pretty annoying to be part of such conversations.

The more common retributive downvoting is, the likelier false positives for "downvoting spats" become, and that will lead to a vicious downward spiral if everyone decides to play tit for tat after someone started it.