(this is talking out of my ass but:)

Variation in genetic robustness/fragility. A known example of this is Iron deficiency in women. If iron in the food is plentiful, no one will notice iron deficiencies, and if it's non existent then everyone will suffer. But if there's an almost sufficient amount of Iron, women will be far more likely to be deficient than men. Women are less robust to lack of environmental iron than men. You can imagine brain development such that everyone has the ability to develop a great brain in a great environment but certain genetics will deal better or worse with certain deficiencies.