If this were true, then the human species would acquire an unequal ratio of men to women (with more women), until the fitness of both was the same (because men would be in greater demand). There are species which work that way, like sea lions. This is known as Fisher's Principle.