would have to be explained by items 4 to 7, which have relatively low weights!

Item 4 is the killer. It is actually heavily weighted in the paper you link, though not in all versions of the SOI. You can score a 50 without ever having had sex or ever expecting to have sex, and up to 32 of those points can come from a response to 4. 4 is obviously not zero sum and need not average out.

There is very good reason to believe that men fantasize about sex more often than women (who buys most pornography?), and there is also reason to believe men are more likely to fantasize about persons other than their current partner. It's also possible that "once per day" is actually relatively infrequent given how often men think about sex. Put these together and it can explain a lot of the variance. The paper does not show breakdown by answers, at least not that I saw.

It's also worth noting they had some significant response-rate issues (RR < .5 for some demographics). If women who like casual sex, are promiscuous, or are prostitutes are less likely to respond than all other groups, that could explain the variance.

Also, there are other versions of the ROI that don't need to average out, as they lump large numbers together (i.e. "More than 8").