Society is thankfully not a zero sum game. In many cases, an immigrant having the option to move to a new country is gaining a significant amount of utility, and the citizens of that country do not lose as much as the immigrant gains (they usually even benefit from the immigrant's presence). And in the cases where the immigrant is taking too much, there already laws in place to counteract antisocial behavior such as stealing or fraud. We already have laws to limit bad outcomes, so restrictions on immigration should tend to cause more harm than good by blocking outcomes regardless of utility. This is in the case where I do not give any advantage to the citizens already in the country by valuing their happiness more, and I don't see a reason why I should.
Tragedy of the commons. On the margin, what a first world society loses and what the immigrant gains from coming to a vastly better country are unbalanced in the favor of the immigrant. On the other hand, unrestricted immigration can lead to cultural shifts, more crime, etc. Eg, if I live in a mansion by myself, and I let someone move in from out on the street, I'm a little worse off and they are vastly better off. If I let every homeless person into my mansion, it very quickly becomes its own slum, I am vastly worse off and on margin each homeless person ...
As Multiheaded added, "Personal is Political" stuff like gender relations, etc also may belong here.