To say "our ancestors did it, and they turned out okay" seems like a general argument against any sort of moral progress.

Not really. It merely means that one's prior should be in favor of the way with a long tradition behind them. This is no more paradoxical then the fact that even though all progress depends on mutations, most mutations are bad. In fact this is merely that principal applied to memetic evolution.