So I checked the average height differences between the sexes, and the male:female ratio is typically between 1.07-1.09. This translates to 8-15 cm of difference. So while it's not as much as 20cm, it's "only" a 2x difference from my prediction. Maybe I'm just bad at translating what I see into centimeters and this difference is much more obvious to you than it is to me.

Maybe gjm's System 1 automatically compensates for the difference -- I know that unless I'm deliberately paying attentìon to people's height I'm much less likely to notice it if a man is six feet tall than if a woman is six feet tall, and for all we know the same might apply to gjm.