First, I want to agree that not every downvote is due to being against group consensus. Sometimes posts (including my own) just aren't well thought out.

I also agree that a downvote only represents one person and not the whole of the LW community. In fact I've thought it would be interesting to be able to see how many upvotes and downvotes each comment has. There's a massive difference between a comment with a score of 0 (or any other score) that got there because 20 people upvoted it and 20 people downvoted it, versus a score of 0 because nobody voted for it at all. The first kind of comment is more likely to be interesting and controversial.

However, I would argue that it doesn't take many people to effect a culture. If there were no moderation, how many trolls do you think it would take to significantly lower the LW quality?

Let's back up and take a macro view (something LWers seem to dislike, but which I think is often necessary.); If posting a non-consensus view tends to result in a negative or 0 karma, and posting a consensus view tends to result in positive karma, then it is likely that people are going to post consensus views much more often than non-consensus views. Even if no one admits it consciously, I would bet the karma system has at least a subconscious effect on what people post.

Because people post consensus views more often, people tend to agree with the consensus more. For example if consensus view has 10 good posts/comments supporting it and nonconsensus view has 0-1 good posts/comments supporting it, then you have seen more and better arguments for the consensus view, and are more likely to believe that.

As a site leans more and more toward a single worldview, people with differing worldviews start actively staying away. If I was a rationalist who rationally thought that the Singularity and FAI weren't the most pressing concerns right now, the chances are low that I would want to join this site. Without having a wide range of views to moderate opinion, consensus becomes like a runaway train becoming progressively more one-sided.

From personal experience, I've known pretty much sense I've joined this site that it's not the sort of place I'd want to stay around. I'll stay around for a month or two, learn what I can from here, and go my merry way. The reason not being random downvotes, which I really don't care about, but rather the resulting culture which I do in fact consider to be an echo chamber that has never learned to argue politely. (In the article I just linked to, I would say that LW is in "Draft 2" stage.)