Warning: broad, slightly unfounded generalizations forthcoming. But I think they're insightful nonetheless.

I think that most people's beliefs are largely determined by reward, power, and status. I want to state explicitly that I don't endorse these social standards, but I think they're pretty solidly established.

For virtually all women, sleeping with multiple men is not high-status. Being with a man who is seeing other women is a marker that she can't get him to "commit" to her, and is therefore somehow deficient. For a substantial majority of men, they are not sufficiently attractive enough (overall, not specifically physically) to entice women into such a lifestyle. In other words, because women feel like they take a huge status hit being with a poly man, your average woman will only consider such a relationship with a man who might otherwise be out of her league. Thus, since most men date women roughly within their own league, most men do not have the opportunity to pursue this.

Men at the top, on the other hand, are probably chasing tail more than they're chasing love or romance. Also, at least based on my knowledge of such men, they don't view female infidelity as being OK - having your woman sleep with other men is definitely a status hit for any man - so it's easier for them to engage in non-consensual non-monogamy than polyamory. This is also "efficient" in the sense that it gives them someone to manage their household/come home to, and some thrills on the side. Polyamory may lack those practical benefits, and is difficult to justify in a social setting for most professions, particularly high-paying ones (that are not entertainment). I would imagine showing up with two dates to business functions, or different alternating dates, might negatively influence one's chance of making partner or the like.

Other obvious obstacles include the legal system and some of the practicalities of child rearing, but I really think the status structure explains a lot of people's reticence to consider the lifestyle. That, and, of course, jealousy.