I'm puzzled by your focus on women. Many men probably don't want to live in a patriarchal society either. I certainly don't.

Thirded. My disquiet comes primarily from the idea of benefiting in status and power from a system that would systematically deny freedom and independent agency and security to half the people I know, and the notion that I would be denied the freedom to take on roles usually assigned to other groups should the situation warrant it.