But they may also contribute to existential risk increase. What sort of calculation have you made that makes you think these people are more likely to contribute to existential risk increase?

I don't think I've seen any reasonable argument that can be made that simply having more random people around will help deal with existential risk. Most likely existential risks (UFAI, grey goo, bioterrorism, whatever) will be caused by people afterall.