Part of the issue is that, even when the hurt is minimal, it's a decision that one's own self-interest outweighs the harm to someone else, and as humans we're not very good at making that calculation objectively.