You do realize, I hope, that there are more than 2 ways for sex to express itself in humans, and humans can have a chromosomal arrangement that is contrary to their phenotypic sex. See XX male syndrome and Androgen insensitivity syndrome for just a couple of the many examples. Admittedly, generalizing from about 99% of the population doesn't seem like too bad of an epistemic move, but it's something to keep in mind.