Women weren't the ones who set up those harems.

Evolutionary fitness is not morality. It doesn't have a thing to do with our preferences. We are adaptation-executers, not fitness-maximizers.