"I think the point is that it didn't evolve to stop us from doing things we consider immoral - it evolved to stop us from getting punished by other people for doing things that are generally considered immoral."

This seems a question begging explanation. It simply pushes the issues back, "why did it evolve just to signal a self-punitive tendency and not to signal a more-adaptive restitutionary tendency?" (Perhaps it only looks more adaptative, but that must be shown.)