Well, quite a lot aren't aware of the existence of polyamory at all. If they think that a person who's in a sexual relationship with someone would necessarily feel betrayed if they knew that person was also having sex with someone else, they would be likely to consider it immoral even without a religious basis.

Numerically, though, I have no idea.

I dunno -- but if you mean “the unexamined remnants of former religiosity” on a societal level¹ rather than on an individual level, then I guess that's the main reason for the overwhelming majority of such people to hold such opinions. There might also be a few people who know that monogamy can curb the spread of STDs and lack a clear distinction between terminal and instrumental values, and/or (possibly incorrectly²) believe that monogamy is “natural” (i.e. it was the norm in the EEA) and commit the naturalistic fallacy, though.

1. i.e., a society used to have a memeplex, originating from religion, which included the idea that “one can only (romantically) love one person at a time”; that society has since shed most of that memeplex, but not that particular idea, which is still part of the intersubjective truth -- even among individuals who were never religious in the first place.

2. “Possibly” meaning that I don't know myself, because I haven't looked into that yet -- not that I've seen all the available evidence and concluded it doesn't definitely point one way or another.