I think rather than going down the Kantian track (deontology is not very convincing to me), I'd take the consequentialist angle:

If most people want to have special status (which seems fairly likely to me, cf the automatic jealousy reflex), you could sometimes end up in a situation where A wants to date B who is in a polyamorous relationship with C as their primary. B is willing to date A as a secondary, so A reasons (wrongly) that they'd rather be B's secondary than not date them at all, and A and B start dating. Eventually after some misery A and B break up when A realises they can't actually take being second-best.

Considering just this argument, polyamory will still be ok if the utility of all the happy polyamorous relationships outweighs the total disutility from all the times when the above scenario happens. I can't say what the frequency or relative utility of either of these situations are, though.

By the way, you are overgeneralizing from your own preferences a bit. Even if you wouldn't be compatible with someone who was genuinely happy with being a secondary, it's not hard to imagine the existence of someone who would be. In which case, as long as all their partners were happy with being secondaries, everything would be ok.