The average size has been going down but the variation has also increased a lot (probably because humans can use culture to stratify by cognitive/physical phenotype). Lord Byron's skull was 2200g (which is downright unwieldy) and he was neurologically atypical. His daughter was more stable (the second X chromosome can offset deleterious stuff in the patrilineal one), but also wrote the first computer program in 1842. There is probably a correlation:

Family History of Psychiatric Disorders Shapes Intellectual Interests (Jan 2012)

Students interested in pursuing a major in the humanities or social sciences were twice as likely to report that a family member had a mood disorder or a problem with substance abuse. Students with an interest in science and technical majors, on the other hand, were three times more likely to report a sibling with an ASD, a range of developmental disorders that includes autism and Asperger syndrome.

Evolution, brain size, and the national IQ of peoples around 3000 years B.C (2010)

Multivariate Cholesky decompositions were performed with each brain volume measure entered first, followed by the four cognitive measures. Consistent with previous research, each brain and cognitive measure was found to be significantly heritable. The novel finding was the significant genetic but not environmental covariance between brain volumes and cognitive measures. Specifically, PIQ shared significant common genetic variance with all four measures of brain volume (r g = .58–.82). In contrast, VIQ shared significant genetic influence with neocortex volume only (r g = .58). Processing speed was significant with total brain volume (r g = .79), neocortex (r g = .64), and white matter (r g = .89), but not prefrontal cortex. The only brain measure to share genetic influence with reading was total brain volume (r g = .32), which also shared genetic influences with processing speed.

The neuroscience of human intelligence differences (2010)

In differential psychology there has been a tradition of seeking fundamental parameters of cognitive processing or single biological variables that might account for intelligence differences. The results have been sparse, but two biological findings have persisted and accumulated: general intelligence differences are substantially heritable; and general intelligence and brain size show modest, positive correlations.