Then what limited the growth of forager peoples so substantially?

I am so glad you asked, because the answer to your question reveals a fundamental misapprehension you have about forager societies and indeed, the structure and values of ancestral human cultures.

The fact is that forager populations don't grow as fast as you think in the first place, and that across human cultures still living at or near forager methods of organization, there are many ways to directly and indirectly control population.

It starts with biology. Forager women reach menarche later, meaning they're not fertile until later in life. Why? Largely, it's that they tend to have much lower body fat percentages due to diet and the constant exercise of being on the move , and that's critical for sustaining a pregnancy, or even ovulating in the first place once you've reached the (much higher) age where you can do that. Spontaneous abortions or resorption of the fetus are rather common. Women in an industrial-farming culture attain menarche quite a bit earlier and are more likely to be fertile throughout their active years -- it only looks normal to you because it's what you're close to. So right out of the gate, forager women are less likely to get pregnant, and less likely to stay that way if they do.

Next biological filter: breastfeeding. Forager women don't wean their children onto bottles and then onto solid food the way you experienced growing up. Breastfeeding is the sole means for a much longer period, and it's undertaken constantly throughout the day -- sleeping with the baby, carrying them around during the daily routine. It goes on for years at a time even after the child is eating solid food. This causes the body to suppress ovulation -- meaning that long after you're technically able to get pregnant again, the body won't devote resources to it. All the hormonal and resource-delivery cues in your body point to an active child still very much in need of milk! Not only that, but it's routine in many such societies for women to trade off breastfeeding duty with one another's children -- the more kids there are, the more likely it is that every woman in the proximate social group will have moderately suppressed fertility. It's a weak effect, but it's enough to lengthen the birth interval considerably. In the US, a woman can have a baby just about every year -- for modern-day foragers, the birth interval is often two to five years wide. It's harder to get pregnant, and once you do, the kids come more slowly.

The next layer is direct means of abortion. In the US that tends to be pretty traumatic if it's not performed by a medical specialist. In some cases it still is for forager women -- the toolkit of abortives across all human cultures is very wide. Midwives and herbalists often have access to minimally-invasive methods, but they also often have painful or dangerous ones. What you won't find is many that are truly ineffective. Methods range from the unpleasant (direct insertion of some substance to cause vaginal bleeding and fetal rejection), to the taxing or dangerous (do hard work, lift heavy objects, jump from a high place) to fasting and ingestible drugs that can induce an abortion or just raise the likelihood of miscarriage.

The last layer is infanticide (and yes, we have this too, though it's a deprecated behavior). In all cultures that practice it it's considered a method of last resort, and it's usually done by people other than the mother, quickly and quietly. Forager cultures are used to having to do this from time to time, but it's still a rare event -- certainly not a matter of routine expedience.

The point I'm making is that population growth unto itself is not a goal or a value of forager societies like those every human being on earth is descended from (and which some still occupy today). Growth, as an ideological goal, is a non-starter for people living this way. Too many mouths to feed means you undercut the abundance of your lifestyle (and yes, it truly is abundance most of the time, not desperate Malthusian war of all against all) -- and forager lives tend to be pretty good on the whole, filled with communitas and leisure and recreation aplenty as long as everybody meets a modest commitment to generating food and the supporting activities of everyday life. I'm not making it out to be paradise; this is just really what it's like, day to day, to live in a small band of mostly close relatives and friends gathering food from what's available in the environment.