A quick google search found this:

Emma Chapman, Simon Baron-Cohen, Bonnie Auyeung, Rebecca Knickmeyer, Kevin Taylor & Gerald Hackett (2006) Fetal testosterone and empathy: Evidence from the Empathy Quotient (EQ) and the “Reading the Mind in the Eyes” Test, Social Neuroscience, 1:2, 135-148, http://dx.doi.org/10.1080/17470910600992239

I can't find a citation for the whole story right now, but as I remember it, it goes something like this: When the first wave of testosterone hits a male fetus, it kills off well over 80% of the brain cells responsible for empathy and reading emotions. Which is not as bad as it sounds, some of them do grow back. And then comes puberty...