If that were true, it would cause my whole system of values to collapse, and it feels like stating an incoherent statement, like "If sexism is morally and scientifically justified, then..."

Really? That doesn't seem incoherent, just incredibly unlikely. Could you elaborate what you mean by "sexism"?

EDIT:

you cannot look into someone's head and say "you're definitely transsexual"

Perhaps someday: transexual differences caught on brain scan - New Scientist

What if the device conflicts their self-reporting? (I suspect I'm anxious about the possibility of it filtering me, specifically.) What should we consider more reliable -- the machine or self-reporting?

Well, I personally would be fine with a non-transexual (by this defenition) getting reassignment surgery and living under a new name ... but I don't think it should be covered by medical insurance, for example. It should be cosmetic lifestyle choice, like a piercing only more so.