if early farming was so much worse than hunter-gathering that we can tell just from the fossils, then why did civilization ever get started? There must have been something compelling or self-sustaining or network effects or something about it.

I tend to think of this by analogy with gene-centered evolution. Just as natural selection selects for genes which are particularly good at reproducing themselves without any special regard for the well-being of their carriers, cultural evolution selects for similarly potent memetic systems without any particular regard for the well-being of the people propagating them.

From skeletal evidence forager lifestyles seem on average a lot healthier, but they also require much lower population densities. You can fit a lot more people per unit area with an agriculturalist lifestyle: if skeletal proxies are to be believed they'll individually be weaker, sicker, and shorter-lived, but they'll be populous enough that the much rarer foragers are going to have trouble displacing them. Cycle that over a few thousand years and eventually civilization ends up ruling the world, with the few remaining foragers pushed into little enclaves where agriculture is unsustainable for one reason or another. We'd occasionally see defections from one lifestyle to the other, but historically they don't seem very common.

The tricky part of this model seems to be figuring out how forager populations self-limit without lowering quality of life to agriculturalist levels. I'm not anthropologist enough to have a definitive answer to this, but I'd speculate that forager resource acquisition isn't as linearly dependent on population as agriculture is: put too many people in a given area and you end up scaring off game, overconsuming food plants, et cetera. Over time I'd expect this to inform territorial behavior and intuitions about optimal group size. Violence is probably also part of the answer.