It's not like we live in a "fair" universe where every male is entitled to a female.

Isn't that the point? Indeed because there's no law of physics preventing more than n men from experiencing long-term involuntary celibacy, since all other things being equal an universe where (n + 1) men out of N experience it is arguably worse than one where n men out of N experience it, then it's up to us to make the former case less likely (unless the costs outweigh the benefits).