There's a perfectly good (IMO) non-cultural reason for that, namely that the average man is physically bigger than the average woman and therefore he's more likely to seriously harm her by hitting her than the other way round.

On the other hand, some women are bigger and/or stronger than some men. Size is hardly a hidden factor. Why not use that instead of a surrogate?