Why even take species-groups instead of groups defined by skin color, weight or height? Why single out one property and not others?

Typically human xenophobia doesn't single out one attribute. The similar are treated preferentially, the different are exiled, shunned, excluded or slaughtered. Nature builds organisms like that: to favour kin and creatures similar, and to give out-group members a very wide berth. So: it's no surprise to find that humans are often racist and speciesist.