Which rather begs the question of why people move to subsistence farming. Perhaps it's a group selection thing

Did you just say the words "group selection"? Out loud?

There's no need to suppose that subsistence farming reproductively benefits the group of farmers but not the individual - that doesn't even seem to make much sense in this context.

I believe I recall seeing other studies claiming that the hunter-gatherer life is being slightly overromanticized; they might have been healthy and tall, but they also underwent a good deal of stress and fear. The carrying capacity of the environment would have been determined by lean years, not average years. One good episode of starvation and you can see why they might want to try raising a few crops. And then anyone who didn't bother raising crops would have been pushed out / exterminated by the far more numerous farmers.