Isn't the usual explanation that "having had a lot of partners" is seen as a positive trait for a man, but a negative one for a woman? (or at least, it's a more desirable/less undesirable trait for a male).
Obviously that's the main implication, and it's very likely a significant cause of the difference. It's not necessarily the only one - and if we're trying to measure that difference, we'll need to take into account the other causes of differences.
From The New York Times:
So how sketchy is the research on human sexual behavior, anyway?