First, I'm going to call them 'N' and 'J', because I just don't like the idea of this comment being taken out of context and appearing to refer to the real things.

Does there exist a relative proportion of N to J where extermination is superior to the status quo, under your assumptions? In theory yes. In reality, it's so big that you run into a number of practical problems first. I'm going to run through as many places where this falls down in practice as I can, even if others have mentioned some.

• The assumption that if you leave J fixed and increase N, that the level of annoyance per person in N stays constant. Exactly how annoyed can you be by someone you've never even met? Once N becomes large enough, you haven't met one, your friends haven't met one, no one you know has met one, and how do you really know whether they actually exist or not? As the size of N increases, in practice the average cost of J decreases, and it could well hit a bound. You could probably construct things in a way where that wouldn't happen, but it's at least not a straightforward matter of just blindly increasing N.
• It's a false dichotomy. Even given the assumptions you state, there's all manner of other solutions to the problem than extermination. The existance and likely superiority of these solutions is part of our dislike of the proposal.
• The assumption that they're unable to change their opinion is unrealistic.
• The assumption that they hate one particular group but don't then just go on to hate another group when the first one is gone is unrealistic.
• The whole analogy is horribly misleading because of all the associations that it brings in. Pretty much all of the assumptions required to make the theoretical situation your constructing actually work do not hold for the example you give.

With this much disparity between the theoretical situation and reality, it's no surprise there's an emotional conflict.