Are you asserting that if I start talking about person X in a conversation... talk about what he does for a living, how he raises his children, how he gets along with his parents and so forth, that a typical listener will reliably understand person X's gender to have not been specified? Will, for example, not be at all startled if I talk about him going in for a gynecological or prostrate exam?

Because I suspect that that claim is demonstrably wrong.

On the other hand, if you're instead claiming that although a typical listener will reliably assume person X is male, they'd be incorrect to do so, because "he" is also a gender-neutral pronoun... well, OK. I won't contest that claim, and I'll agree that it matters in any situation where I'm not primarily interested in the actual meanings that get reliably communicated to other people by my speech.