The problem is that it's a two-way street. What you see is not what I see, and if person X is claiming A, I myself might be seeing D instead. Reality-wise, we would probably have to check their genes, but that's less practical. So instead people go to their intuition and just call them whatever they think fits.

It's also a pretty big offense to call a woman a man, and vice-versa, so there's still risk in that.