I will bite that bullet. Actually yes they should have! Since segregationists where right about specific undesirable consequences of integration that could have been avoided with a better thought out approach or more modest goals. Indeed very basic segregationist arguments against social engineering measures that where undertaken such as forced busing are surprisingly hard to beat.

Now obviously being against such invasive social engineering or affirmative action or disparate impact doctrine is also a possible principled libertarian stance but the result is segregation so segregationists often made those arguments as well and often made them well. They where engaged in motivated cognition finding the best possible reasons against a policy just as many people where engaged in motivated cognition to find the best possible reasons for policies. You need to set up a system where those offset each other as much as possible if you want to be confident in your epistemology. If you don't you are just writing the bottom line first and then generating the system that comes to the conclusion you want.

If you are a normal educated Western person, you have probably never read (certainly not in the course of a normal education) a non-straw-man argument against women's suffrage, for eugenics, against parliamentary democracy or nearly any other kind of social political change our society has done for the past several centuries.

This should scare you unless you believe society without much well informed designing happens to function very much like a FAI when editing our instrumental and terminal values in unpredictable ways.