For the purposes of a all women college you have to ask yourself about the purpose of limiting the college to women.

Maybe there a perception out there that math isn't a women's subject. In mixed-gender classes woman are more likely to spent effort into signalling their strong femininity. The are spending more effort into engaging in actions that signal high mating market value when suitable mates are around.

Do transwomen trigger the same behavior? I don't think that the gender that's assigned at birth matters. On the other hand "identifies" is a complex word. Just checking a checkbox isn't enough to stop being seen as a potential mating partner. The transwomen who spends enough effort on looking female that strangers easily identify them as being female on the other hand is unlikely to trigger mating market behavior.

Another reason to limit the college to women is about women being a minority that's discriminated against. Transwomen do get discriminated against them and don't get included in old boys networks. From that perspective it also seems fine to accept transwomen.

Maybe you can also think of other reasons for the policy of being all-women. Check whether those reasons matter for transwomen.

There seem to be strong laws against gender discrimination in the US, how does your college avoid getting sued for discrimination?