At some point we really do have to enforce community norms to prevent the level of discourse from deteriorating. Antisocial and obnoxious behavior are perfectly valid reasons to downvote a comment, I have a hard time believing you really think they aren't, and I'm reasonably confident that most of LW is okay with the idea, judging by other comments I've seen receive downvotes (not to pick on him, but Annoyance has gotten some of this) for no obvious reason other than tone.

The issues are where to draw the lines, and how to handle it when the community is sharply divded on what constitutes "polite, respectful discussion". Frankly, the main person I see drawing "us vs. them" lines here is you.