It's not just that the risk of defection increases with the number of people, though that's certainly true. It's also that the dynamics for 3+ people are qualitatively different from the dynamics for two people -- if you're in a monogamous relationship, you can't play the other people off each other to your benefit, because there's only one other person.

(This is also technically true for couples with children, but children are probably less likely to be skilled at and inclined toward that sort of manipulation than polyamorists, and married parents (though IME not divorced ones) should be able to put up enough of a united front to prevent the obvious failure modes.)