Because we evolved to care about things like fairness in an environment where everyone knew each other, and if you cheated someone, everyone else in the village knew it. And, modern humans still employ their evolved instincts. Therefore, agents who lack moral concerns can exploit the fact that humans are using intuitions that were optimized to work in a different situation. For instance, they can avoid doing things so heinous that society as a whole tries to hunt them down, and once they have exploited someone, they can just move.