For a substantial majority of men, they are not sufficiently attractive enough (overall, not specifically physically) to entice women into such a lifestyle. In other words, because women feel like they take a huge status hit being with a poly man, your average woman will only consider such a relationship with a man who might otherwise be out of her league.

Another thing that happens extremely often: A male who is already sexually active with multiple partners just appears a whole heap more attractive so women end up in a relationship with them more readily. If you have a couple of partners already you have to work hard to stop yourself from getting more.