But then, why don't people downvote more often? Especially if downvoting is completely anonymous and requires minimum effort?

One guess is that downvoting is too simple; it does not pattern-match to a personal attack, therefore it does not bring the related emotions.

Other guess is "in group / out group" distinction, where people on the same site are percieved as members of the same subgroup, and you don't want to make your subgroup weaker. But then why does the same effect not stop trolling? Do trolls percieve themselves as members of a dominant subgroup inside the weaker subgroup, showing them who is really the boss? (The imaginary dominant subgroup = people who don't care about this website.)