I'm afraid I don't really have a good answer. I think that where we draw our sphere of moral concern is basically arbitrary; I just happen to find the idea of sacrificing human welfare for non-human animals deeply unattractive. I might be willing to accept a lexicographic ordering that took other animals' welfare into account only when human well-being was unaffected, but I doubt that adopting such an view would have (m)any practical consequences.
I agree with everything you said, except that I believe non-human animals deserve non-zero moral weight.
Do you believe infants, or people with dementia, or severe mental disabilities deserve non-zero moral weight? Independent, of course, of how their welfare effects the welfare of mature intelligent humans who care about them.
Does witnessing animal torture not bother you?
http://www.youtube.com/watch?v=7eQQQBn4dlo&feature=channel
Note, I do not endorse everything PETA does.
I have noticed that among philosophers, vegetarianism of one form or another is quite common. In fact, I became a vegetarian (technically a pescetarian) myself partly out of respect for an undergraduate philosophy professor. I am interested in finding out if there is a similar disproportion in the Less Wrong community.
I didn't request that this go into Yvain's survey because I want more information than just what animal products you do or don't eat; I'd also like to see nuances of the reasons behind your diet. There are a lot more shades than carnivore/vegetarian/vegan - if you want to be a vegetarian but are allergic to soy and gluten, that's a compelling reason to diversify protein sources, for instance. I'd also like to hear about if you avoid any plant foods (if you think they're farmed in a way that's environmentally destructive or that hurts people or if you have warm fuzzy feelings for plants, maybe). Here are some questions that come to mind: