The most likely cause of the shrinkage, in terms of selective pressure, is that maternal mortality and starvation were big killers, and being smart didn't help much when it came to not dying from them.
If this were the case, how would they have gotten large in the first place? What would have changed to make starvation and maternal mortality larger killers than they were earlier?
"Agriculture" for the first. For the second, maternal mortality has always been the main limit on human cranial size, Giving birth is ridiculously dangerous to human beings compared with other mammals, and our infants are much less developed, both of which is directly linked to the fact that our heads are such outliers in size, and there is in addition a whole host of finicky adaptions to accommodate the largest head possible. - the infant soft spot in the skull, the extra wide pelvis.. There must have been really absurdly strong selection pressure for "bigger brains" for a long time to create all that, and it pushes the envelope on what is biologically practical for our bodyplan quite hard.
So we say we know evolution is an alien god, which can do absolutely horrifying things to creatures. And surely we are aware that includes us, but how exactly does one internalize something like that? Something so at odds with default cultural intuitions. It may be just my mood tonight, but this short entry on the West Hunter (thanks Glados) blog really grabbed my attention and in a few short paragraphs on a hypothesis regarding the Hobbits of Flores utterly changed how I grok Eliezer's old post.
I have to break here to note that was the most awesome fact I have learned in some time.
That last sentence just struck me with utter horror.