He didn't make the argument that simply having girls present is only a mild distraction. Instead, his argument is that one should accept distractions because they are present in normal life.

I know, I agree with your argument. (Without supporting sex segregated lesswrong meetups as being a remotely practical idea!)