Such groups are also in the most marginal territories, that we haven't pushed them out of, like the !Kung. Any observed shrinkage might be environmental (eg. nutritional) compared to the old skulls we are looking at, from richer and more normal temperate areas.