You assert these things very confidently, but without any evidence. How exactly do we know that this state of
affairs existed in human prehistory?
Archaeological evidence regarding the health and population density of human beings and their dietary habits. Inference from surviving examples. The null hypothesis, that we didn't start with agriculture and therefore must have been hunter-gatherers for most of our existence as a species. The observatiion that the traits generally associated with the Malthusian trap are common experiences of agricultural societies and dependent upon conditions that don't obtain in predominantly and purely hunter-gatherer societies.
This, however, provides no answer to the question why individuals and small groups wouldn't defect, regardless of the subsequent collective consequences of such defection.
They might defect, but it'd gain them nothing. Their cultural toolkits and food-gathering strategies were dependent upon group work at a set quota which it was maladaptive to under- or overreach. An individual can''t survive for long like this compared to a smallish group; a larger group will split when it gets too big for an area, a big group can't sustainably form.
How can a society, i.e. a group, have "values" and "incentives," if you're not postulating group selection?
The answer to this lies in refuting the following:
As soon as even a small minority of the forager population starts cheating and reproducing above the replacement rate (by evolving either cultural memes or hereditary philoprogenitive behaviors that motivate them to do so), in a few generations their exponential growth will completely swamp everyone else.
"A small minority of the forager population" has to be taken in terms of each population group, and those are small. A small percentage of a given group might be just one or two people every handful of generations, here. A social umbrella-group of 150 scattered into bands of 10-50 throughout an area, versus just one or two people? Where's the exponential payoff? The absolute numbers are too low to support it, and the defectors are stuck with the cultural biases and methodologies they know. They can decide to get greedy, but they're outnumbered by the whole tribe, who are more than willing to provide censure or other forms of costly social signalling as a means of punishing defectors. They don't even have to kill the defectors or drive them out; the defectors are critically dependent on the group for their lifestyle. The alternatiive will be unappealing in all but a vast majority of cases.
You need the kind of population densities agriculture allows to start getting a really noticeable effect. It's not to say people don't ever become tempted to defect, but it's seldom a beneficial decision. And many cultures, such as the San ones in South Africa, have cultural mechanisms for ensuring nobody's ego gets too big for their britches, so to speak. Teasing and ribbing in place of praise when someone gets a big head about their accomplishments, passive reminders that they need the group more than they individually benefit it.
This isn't so much about group selection,as it is about all the individuals having their raft tied to the same ship -- a group big enough to provide the necessities of life, which also provides a lot of hedonic reinforcement for maintaining that state of affairs, and a lot of non-coercive negative signalling for noncompliance, coupled with the much more coercive but morally neutral threat presented by trying to make a living in this place all by yourself.
If you break a leg in a small group, the medical practitioner splints it and everyone keeps feeding you. If you do that by yourself, it probably never heals right and the next leopard to come along finds you easy pickings. That's what defection buys you in the ancestral environment.
a larger group will split when it gets too big for an area
Say there are two kinds of forager groups, one which limits reproduction of its members by various means, and another that does not limit reproduction and instead constantly grows and splits and invades other groups' territories if needed. Naively I would expect that the latter kind of group would tend to drive the former kind out of existence. Why didn't this happen?
From Mike Darwn's Chronopause, an essay titled "Would You Like Another Plate of This?", discussing people's attitudes to life:
Conclusion, graphs, and references in article. As usual, I recommend reading Chronopause.com as Darwin has many good articles; to quickly link a few: