Even if such women are numerous enough, a socially awkward man who is bad at reading others will not be able to reliably identify them, and so will occasionally creep out more-averse women, and that may be enough for him to be banished from the whole social group. This is a matter for empirical measurement.