In the sociological "let's all decide what norms to enforce" sense, sure, a lack of "morality" won't kill anyone. But in the more speculative-fictional "let's all decide how to self-modify our utility functions" sense, throwing away our actual morality—the set of things we do or do not cringe about doing—in ourselves, or in our descendants, is a very real possibility, and (to some people) a horrible idea to be fought with all one's might.

What I find unexpected about this is that libertarians (the free-will kind) tend to think in the second sense by default, because they assume that their free will gives them absolute control over their utility function, so if they manage to argue away their morality, then, by gum, they'll stop cringing! It seems you first have to guide people into realizing that they can't just consciously change what they instinctively cringe about, before they'll accept any argument about what they should be consciously scorning.