I'm going to ignore the Elitist Jerks article if you don't mind, because really the only association it has with the topic at hand is the name of a community in no way affiliated with Less Wrong, and moreover a casual visitor is unlikely to ever read it.

Besides that, you are mainly seeing reactions to your own posts related to elitism, and I think you're misreading those reactions. In particular, other people don't read comments on the "pro-or-anti-elitist" scale, and their upvotes and downvotes don't reflect that. (See here for why I think your post was downvoted.)