I think there may also be a difference in time: Contrarian posts and comments will sometimes get voted down very quickly, within a few minutes of being posted, then be voted up over the next few days. It could be that LW conformists are more likely to check the site obsessively, and so most of the first dozen people to see a comment are conformists.

I attribute this to disagreement within the immediate discourse. In general whenever a group of a few people are having an argument in a sequence of comments and (at least) one side clearly cares a new comment that refutes the previous one often receives quick downvotes. This is independent of whether the position disagrees with the community at large and depends directly on the 'other side' taking it personally.

I know that some of my highest voted series of comments actually started at below -2. It is only after a day or two that they settled at their stable 'approved of' status. In contrast to your suggestion this would seem to suggest that it takes time for the 'mainstream' tide of opinion on comment value to overwhelm the eddy current of personal dislike.