The notion that our brains just got more efficient and 'therefore' could shrink in size appears very shaky to me. This 'therefore' comes from fallacy of anthropomorphizing the evolution. Evolution doesn't work to a goal of optimizing some sub-unit in the organism while preserving specifications, in the way that a team on an engineering project would.

Suppose that there were mutations which improved brain efficiency, but had other unfortunate side effects - these would be adaptive above a certain brain size threshold. We shouldn't of course presume that this was the case, but it's not implausible either.