This is an entirely reasonable concern. But the comments are generally the highest quality I can think of on almost any forum. So I'd say it's certainly not broken.
Comment quality was approximately as high on Overcoming Bias before the founding of Less Wrong, and there was no voting on OB.
Voting did greatly reduce the need for editors to moderate comments. In particular, part of the reason that comment quality was extremely high on Overcoming Bias is that Eliezer moderated comments on Eliezer's posts and Robin moderated comments (with a heavier hand in my experience than Eliezer did BTW) on Robin's posts, and if voting had not been introduced (by Eliezer, the Future of Humanity Institute and Tricycle Development) Eli...
Eliezer Yudkowsky has passed an arbitrary milestone: 100,000 karma points on Less Wrong. Allow me just a moment to celebrate this like we celebrate other arbitrary milestones, like birthdays.
I think that Eliezer's karma score vastly under-rates his relative contribution to this site. For example, his score is only about 13x my own score, but I think it's obvious he has contributed far more than 13x as much value to this community as I have.
This is probably due to the fact that good posts today get far more upvotes than earlier good posts, when I suspect the community was smaller. For example, my rather simple and insignificant post Secure Your Beliefs received 34 upvotes, which is more than almost any of Eliezer's epic and brilliant posts of the past have received, for example Terminal Values and Instrumental Values.
So at this arbitrary milestone, I'd just like to say a quick word to Eliezer:
Thanks.
You've done a lot.
Okay, that's all! I hope this doesn't come across as "sucking up to the Dear Leader," but instead as the sincere appreciation it is. There is a good reason I list Eliezer as one of my heroes-even-though-we-shouldn't-have-'heroes' over here.