Sure, saying women are more likely to get breast cancer isn't sexist; but this is a safe example. What if we had hard evidence that women are less intelligent? Would it be sexist to say that, then? (Any objection that contains the words "on average" must contend with the fact that any particular women may have a breast cancer risk that falls anywhere on the distribution, which may well be below the male average.)

I wouldn't say it is, but other people would use the word “sexist” with a broader sense than mine (assuming that each person defines “sexism” and “racism” in analogous ways).