The notion that our brains just got more efficient and 'therefore' could shrink in size appears very shaky to me. This 'therefore' comes from fallacy of anthropomorphizing the evolution.

Up voted for taking evolution seriously when it tells us that it simply wants to maximize fitness and dosen't care about what we like.

I think you are right on our biases regarding this.