flipped the secondary sexual characteristics in my brain
This is a potentially valid but minority interpretation of the statement "X is trans.", held by no trans people I have discussed the subject with (and it tends to come up, eventually, across a sample size of a dozen middling-to-close friends). The more common one, which all trans people I know hold, is "At some point during my development I gained a strong repulsion from the particular cultural bundle labeled with the gender which matches the dominant one for my sex." Generally this is also a strong attraction to the other major gender bundle, but sometimes the trans person finds that bundle equally off-putting and rejects both, and others exchange between the two regularly or hold themselves to be both.
Also, the poor coding practices which evolution uses make this more, not less, likely to occur. Most people who study trans issues, history, etc. agree that there were probably high rates of 'masked' trans people in the past, who kept their personal identification preferences secret.
"At some point during my development I gained a strong repulsion from the particular cultural bundle labeled with the gender which matches the dominant one for my sex."
There was once an old Saturday Night Live skit about nationality-change operations. But given your explanation, wouldn't that imply that some people might actually think of themselves as a different nationality than the one they were born into? It's certainly a cultural bundle, after all. Why don't we see that?
Rationality quotes time!
The usual rules: