You will save an expected number of animals equal to the number of animals you don't eat that you would otherwise have eaten. You might not personally tip any balances, because factory farms operate on large scales; but you might be the Nth vegetarian whose decision justifies shutting down a factory farm full of suffering animals. The utility of the latter counterbalances its small likelihood.
Also, stigma? Where do you live? If anything, being a vegetarian lets me be smug and self-righteous in social situations.
Perhaps my social circles are unusual, but in my experience smug self-righteousness tends to have some stigma associated with it.
I have noticed that among philosophers, vegetarianism of one form or another is quite common. In fact, I became a vegetarian (technically a pescetarian) myself partly out of respect for an undergraduate philosophy professor. I am interested in finding out if there is a similar disproportion in the Less Wrong community.
I didn't request that this go into Yvain's survey because I want more information than just what animal products you do or don't eat; I'd also like to see nuances of the reasons behind your diet. There are a lot more shades than carnivore/vegetarian/vegan - if you want to be a vegetarian but are allergic to soy and gluten, that's a compelling reason to diversify protein sources, for instance. I'd also like to hear about if you avoid any plant foods (if you think they're farmed in a way that's environmentally destructive or that hurts people or if you have warm fuzzy feelings for plants, maybe). Here are some questions that come to mind: