I am curious as to whether drawing attention to the author's gender is purely undesirable, or only undesirable where that gender already makes up a substantial majority of the readership/authorship.

But you weren't speaking in terms of the author's gender. The preceeding sentence ends with "(...) no matter what you say or think about them.", creating a second-person context, hence the implication of projecting the author's gender onto the audience.

If you had phrased the following sentence in first person, or as an acknowledged-to-be-male third person, it likely would have bothered people less.