That objections disappears if you assume an equal gender ratio in the population at large, which is usually the case.
Even in this case, you could have a minority of men who have sex with 100 women very quickly, then die, surrounded by other men who have sex with either 0 or one woman but live for a long time. Women in this scenario all live a long time. Then the average for men will be lower than the average for women.
From The New York Times:
So how sketchy is the research on human sexual behavior, anyway?