Then the right meta thing we need to do is: remind people that their (not) downvoting (or even upvoting of things they consider wrong, but feel it is fair to give them visibility because of some abstract principle) has consequences. Most obviously, the consequence of getting more of what they don't downvote, or even upvote.

Next thing would be to emphasise that there are different forms of criticism, and that well-written criticism is typically well accepted (see Holden). Therefore we do not have to pay attention to trolling-style criticism.

Then, if someone sees a comment saying "okay, I know this is stupid and offensive, and was already said hundred times, but I upvoted it anyway because I think LessWrong needs dissenting voices", it can be responded by a link to that article.