Nobody needs to reference anything religious in order to be be tempted as the receiver of the attack to label it wrong. "Geez this feels really uncomfortable but I guess I can't judge this because I don't have any moral authorities to tell me that would be okay". That would be like arguing that soldiers would be physically unable to shoot unless some general orders them to. While it is true that soldiers under a command of a general will wait till an order to shoot, a soldier that knows they are generalless will not wait for external ques to act. Correspondingly humans are capable of being independent moral agents. They don't need to be told to be moral. Even if their goodness can be boosted by being part of a communal moral discussion.