This is a great point, and was advanced by Steven Davis, who claimed that eating grass-fed cows would cause fewer deaths than being vegetarian. Matheny however found an error in his calculation - you can see the paper for full details, but the short version is that veg diets cause much less harm.