I think there may also be a difference in time: Contrarian posts and comments will sometimes get voted down very quickly, within a few minutes of being posted, then be voted up over the next few days. It could be that LW conformists are more likely to check the site obsessively, and so most of the first dozen people to see a comment are conformists.
I attribute this to disagreement within the immediate discourse. In general whenever a group of a few people are having an argument in a sequence of comments and (at least) one side clearly cares a new comment that refutes the previous one often receives quick downvotes. This is independent of whether the position disagrees with the community at large and depends directly on the 'other side' taking it personally.
I know that some of my highest voted series of comments actually started at below -2. It is only after a day or two that they settled at their stable 'approved of' status. In contrast to your suggestion this would seem to suggest that it takes time for the 'mainstream' tide of opinion on comment value to overwhelm the eddy current of personal dislike.
My impression is that critiques of lesswrong mainstream positions and arguments for contrary positions are received well and achieve high karma scores when they are of very high quality. Similarly posts and comments that take lesswrong mainstream positions will still be voted down if they are of very low quality. But in between there seems to be a gulf: Moderately low quality mainstream comments will stay at 0 to -1 karma while contra-mainstream comments of (apparently) similar quality score solidly negative karma, moderately high quality mainstream comments achieve good positive karma while similar quality contra-mainstream comments stay at 0 to 2.
Do you share my impression? And if this is the case, should we try to do something about it?