I disagree with cosmopolitanism when it comes to "not harming" as well. I think needlessly inflicting suffering on human beings is always really bad, but it is worse if, say, you do it to your own children rather than to a random stranger's.