Humans used to consider the other tribe over the hill unworthy of moral consideration because they were "other." You can use "other" as a criterion, but you're increasing, in some small way, the chance that others will use that criterion to avoid giving consideration to you or people you care about.

I don't think "other" is the main criterion either - if we visit another planet and find it inhabited by aliens with approximatively 19th century europe technology, and consider them unlikely to harm us (their planet has no uranium, they're two feet tall and not particularly warlike, and we have nanoweapons and orbital lasers), I would still consider it very immoral to kill one of them, even though they are very "other", even less related to us than broccoli is, and being nice to them isn't particularly in our interest.