Can you put forth a actual reason why men, let alone trans women, shouldn't go in women's bathrooms that doesn't devolve into victim blaming? Did we start posting guards at bathroom doors to check peoples genitals when I wasn't looking?

Thanks for this question! I really tried to find an actual reason (without the stop signal answers like "culture" or "religion") and found following (it's probably highly flawed, but at least worked for me):

Clothes were invented for protection and convenience -> civilization development created social disparity -> poor and slaves couldn't afford clothes -> nakedness ≈ slavery and slaves were the easiest source of sex -> it's shameful and dangerous to be naked -> shame and abscense of genitals around free people sexualized genitals -> slavery slowly becomes illegal in the vast majority of cultures -> majority of people already conditioned that naked opposite genitals ≈ sex, nudists are minority and not mainstream -> with such conditioning (or simply, culture) we can't have common bathrooms. 

Btw, prisoners rape prisoners regardless of genitals.