I wasn't kidding when I said one of the motivations was a desire to not live with large numbers of strangers. One issue might be cognitive load-- the strangers have unfamiliar customs (is a sincere apology accompanied by a smile or a serious expression?) and possibly an unfamiliar language.

As far as I can tell, the economic side of not wanting immigrants is a sort of merchantilism-- a belief that all that matters is where the money is, so that new people showing up and getting paid for work just seems like money getting drained away. Weirdly, rich people who show up and spend money without working locally may be disliked, but they don't seem to be as hated as poor people who do useful work. I don't think it's just about competition for jobs.