The non-caricatured version of Larry Summers opinion was not literally insane.

Let's not forget that it even wasn't his stated opinion. It was one of the three hypotheses he suggested as worth testing experimentally:

1. The high-powered job hypothesis
2. Different availability of aptitude at the high end
3. Different socialization and patterns of discrimination in a search

So more precisely, you can be fired even for a non-mainstream opinion you don't express, as long as people try hard enough to assign it to you connotationally. When the social norm is to disbelieve X, then saying "we should test X experimentally" suggests that you are not disbelieving enough; otherwise you would consider experimentally testing X a waste of time.