You will save an expected number of animals equal to the number of animals you don't eat that you would otherwise have eaten. You might not personally tip any balances, because factory farms operate on large scales; but you might be the Nth vegetarian whose decision justifies shutting down a factory farm full of suffering animals. The utility of the latter counterbalances its small likelihood.

Also, stigma? Where do you live? If anything, being a vegetarian lets me be smug and self-righteous in social situations.