It is true that you receive dimishing marginal returns whenever you try to import people. Even if we were to use the largest available population sink, eventually we'd run into limits. The larger the population sinks you use, the less it has been filtered, so while your returns diminish more slowly, the effort required at the outset is larger.

Given the small size of our group, physics departments are more than large enough population sinks for the forseeable future.