As human population densities increased and complex societies formed, selection pressure for social skills increased, and social skills became more relevant than intelligence. Larger brains usually have fewer long-range connections but more local connections, and long-range connections enable the rapid processing required for socialising. People with autism tend to have larger brains than those without and females tend to have smaller brains than males, so an inverse correlation between brain size and social skills would not surprise me.