There have been other changes as well, which don't fit this generalization. For instance, we now treat the people who do have moral worth much better, in many ways.

Also, there have historically been major regressions along the "percentage of society having moral worth" scale. E.g., Roman Republican society gave women, and all Roman citizens, more rights than the post-Roman Christian world that followed.

Finally, "not random drift" isn't the same as "moving towards a global singular goal". A map with fractal attractors isn't random, either.