Maybe. Maybe society would create new norms to fix that.

I'd like to mention that I'm emphatically not a libertarian (in fact identifying as socialist), and find many absurdities with its basic concept (see Yvain's "Why I Hate Your Freedom); however, I'd always like to learn more about how it could plausibly work from its proponents, and am ready to shift towards it if I hear some unexpectedly strong arguments.