Sam Harris has an argument against absolute moral relativism. There really are absolute moral relativists out there, who say that any moral code is as good as any other, and no one should think poorly of Jeffrey Dahmer because he likes to murder men and screw their corpses.

The way you put it obscures one extremely important difference, namely that between individuals who behave in ways that could never be a general norm in a stable and functional society and societies that are functional and stable even though their norms are extremely different from ours. As far as I see, the supposed relativists who wish to excuse Jeffrey Dahmer are just a conveniently ridiculous strawman for cultural relativism that applies only to other functional and stable societies distant in space or time, which is much more difficult (if at all possible) to refute.

Now you say:

If you can't pass the bar of saying "hurting people is bad", then you shouldn't be allowed to help work out the social contract. We should all be able to agree that hurting people is bad.

But in fact, there's going to be plenty of hurting in any realistic human society. Attempts to argue in favor of an ideology because it has a vision for minimizing (or even eliminating) hurting get into all the usual problems with utilitarianism and social engineering schemes, both theoretical and practical.