"And then anyone who didn't bother raising crops would have been pushed out / exterminated by the far more numerous farmers."

Hunter-gatherers were far better nourished than their farming counterparts, who had plentiful calories but a very limited diet. Once the population rose to the new carrying level, the farmers didn't even have plentiful calories, and rapidly also became subject to starvation in lean years.

In both conditions, the key seems to have been to keep the population well below the maximum in order to ensure a high quality of life. Societies that didn't do this were miserable, but between population pressure and disease managed to exterminate / extirpate the h-gs.

Humanity began to switch from a K to an r reproductive strategy, with obvious consequences.