It looks to me as if you are mixing up a number of different things (what makes someone male or female, versus what constitutes sufficient evidence to treat them so in a given case; what I think their gender is, versus what I would treat it as in a given difficult situtation; etc. I will try to disentangle these things.

The position I am defending here is as follows. (Individual points numbered for cross-reference.)

[EDITED to stop LW's comment formatting messing up my numbers and to complete something I carelessly left unfinished after editing other bits.]

0. There is no single fact-of-the-matter about a person's gender in general, because different notions of gender are appropriate in different circumstances. 1. Of course, for the great majority of people all reasonable such notions coincide; the questions here are about cases where they diverge. 2. For most purposes the best notion of gender is largely a matter of (a) internal mind-state and (b) social role occupancy. 3. The relevant internal mind-state doesn't change rapidly; social role occupancy can in a sense change quickly but evidence of it accumulates more slowly. And of course anatomy and chromosomes and whatnot are even harder to change.

4. In many cases, if someone claims that their gender is not as it superficially appears, the best policy is to believe them. (Note: this is not only about trans people. There are people who are anatomically, chromosomally and hormonally female but look very much like men unless you take their clothes off.) 5. In many others (typically distinguished from those in #3 by the consequences being worse if you take them at their word and they're lying) the best policy is to require stronger evidence of 2a and/or 2b (e.g., legal name change; evidence of having been consistently self-describing as female for some time; testimony of a psychologist who has examined them). 6. In some others (e.g., medicine, major sporting contests) 2a and 2b may be pretty much irrelevant and the only important thing may be genes or gross anatomy.

7. Every possible policy will make some mistakes, with the boring exception that if you define gender by easily visible external features then the policy of using those easily visible external features will not make mistakes. (But either you can't execute that policy without looking in people's pants, or else you will classify some people with female internal anatomy and chromosomes as male.)

So. Can someone's gender, in my view, change on a whim? No (see #3). Can something they do on a whim suffice to make me treat them, at least provisionally, as of one gender rather than another? Yes, but only in "low-stakes" cases (see #4). Does this mean that if everyone thought as I do then our nations' women's restrooms would be flooded with men claiming to be women in order to assault or harass? No, because in higher-stakes cases I would be more cautious (see #5), and in any case the available evidence strongly suggests that even laws that straightforwardly let anyone use any restroom do not cause any increase in such crimes.