This is a great argument, and is known as the "Logic of the Larder" (for reasons I have never comprehended). This paper goes into more detail than you probably care about; the main point is that your guess:

the animal population would likely be greatly diminished

Isn't generally true, because wild animals have a much greater density than farm animals.