The average size has been going down but the variation has also increased a lot (probably because humans can use culture to stratify by cognitive/physical phenotype). Lord Byron's skull was 2200g (which is downright unwieldy) and he was neurologically atypical. His daughter was more stable (the second X chromosome can offset deleterious stuff in the patrilineal one), but also wrote the first computer program in 1842. There is probably a correlation:
Family History of Psychiatric Disorders Shapes Intellectual Interests (Jan 2012)
Students interested in pursuing a major in the humanities or social sciences were twice as likely to report that a family member had a mood disorder or a problem with substance abuse. Students with an interest in science and technical majors, on the other hand, were three times more likely to report a sibling with an ASD, a range of developmental disorders that includes autism and Asperger syndrome.
Evolution, brain size, and the national IQ of peoples around 3000 years B.C (2010)
Multivariate Cholesky decompositions were performed with each brain volume measure entered first, followed by the four cognitive measures. Consistent with previous research, each brain and cognitive measure was found to be significantly heritable. The novel finding was the significant genetic but not environmental covariance between brain volumes and cognitive measures. Specifically, PIQ shared significant common genetic variance with all four measures of brain volume (r g = .58–.82). In contrast, VIQ shared significant genetic influence with neocortex volume only (r g = .58). Processing speed was significant with total brain volume (r g = .79), neocortex (r g = .64), and white matter (r g = .89), but not prefrontal cortex. The only brain measure to share genetic influence with reading was total brain volume (r g = .32), which also shared genetic influences with processing speed.
The neuroscience of human intelligence differences (2010)
In differential psychology there has been a tradition of seeking fundamental parameters of cognitive processing or single biological variables that might account for intelligence differences. The results have been sparse, but two biological findings have persisted and accumulated: general intelligence differences are substantially heritable; and general intelligence and brain size show modest, positive correlations.
The human brain volume has been shrinking over the past 20 000 years or so, after millions years of increase in volume. Not just the brain size, but the brain size relatively to body size as well. We are lacking a tennis-ball sized piece of our earlier brain (and it might even be God-shaped).
Brain is expensive in many ways: energy consumption, birth complications and locomotion impairment for females, lower survival of head impacts i'd guess. The damn thing along with supporting structures is heavy and awkwardly located, etc.
And the big brain can only be advantageous if it improves reproduction substantially, with larger brained individuals being sufficiently more successful at surviving and reproducing than smaller brained individuals, as to negate the above-mentioned cost.
That must have been the case through the evolution up to a couple tens thousands years ago, to produce the big brains that we have. It is clear to see that in past 20 000 years, the environment in which humans live has undergone very significant change due to emergence of societies; the new environment may not be pushing us as hard [in the direction of intelligence], at least on the individual level. [and may have been pushing us too hard for smaller brains, thanks Nornagest for making that point]
We were evolving ability to think, until it got just about to the point of being barely able - with great difficulty and many falls - to think useful thoughts. If we were species that were evolving flight, we'd be the species that could just barely fly, and recently flew over a river, entering new land. In the new land, everything is different. And our wings were shrinking at very rapid rate.
The important question is - Did we lose any functionality since then? Are we dumber? Are we less sane in some way? (The palaeolithic humans did not seem to do any really insane religious stuff)
The notion that our brains just got more efficient and 'therefore' could shrink in size appears very shaky to me. This 'therefore' comes from fallacy of anthropomorphizing the evolution. Evolution doesn't work to a goal of optimizing some sub-unit in the organism while preserving specifications, in the way that a team on an engineering project would.
The optimization could as well instead make brain even larger, if said improvement made larger brain pay off more. One would have to show that some improvement in brain efficiency has actually decreased advantage of big brains over small brains, to explain the smaller brains with them being more optimized.
The evolution optimizes the whole organism, not the brain, and there's very many of other factors that have changed at that specific time that may as well have decreased selection pressure towards intelligence or increased the costs.
In my opinion the sensible default hypothesis should be that we had a decrease in some functionality, and likely are still declining.
My best guess is that it is the capacity to invent solutions on spot and think by ourselves, that we are losing. Before emergence of societies, the technological progress was severely limited by information loss. Any smart individual could massively improve fitness of the relevant genes by (re)inventing some basic, but extremely effective techniques, which he'd teach mostly to genetically related individuals. The technique would easy become lost, creating again an opportunity for intelligence to succeed - reinventing it.
Even very simple invention requires massive search in the vast space of possibilities. Precisely the kind of task that one would expect to benefit from larger raw computational power.
edit:
Some clarification with regards to the need for innovation. In the long run, it is not enough to just do what you're taught. Teaching is a lossy process. You need to improve upon what was taught to you a little to make the tool as good as your ancestor made - you need minor innovation to merely preserve the tools - a little more innovation and you'll improve over time, a little less and you'll lose it over time. The little children have to figure out everything from a few clues; they don't download some braindump of the wisest elder to be able to speak, they essentially figure out an alien language - a very difficult task.