I've read the comments, and nobody seems to have mentioned the different functions of speech, and the different goals of limitations. Yvain's examples of Holocaust denial and "shouting fire" are not the same sort of thing.

Speech as an act causally linked to a desired consequence unrelated to communication, like falsely shouting "fire" in a crowded theater, or commanding a subordinate to shoot someone, or etc., isn't really protected (I'm not very familiar with US Law, but I can't think of counterexamples). These limitations don't usually register as "censorship".

Speech as transmission of information is protected, but with many [mostly] uncontroversial exceptions, like privacy, intellectual property, and state secrets.

Most controversies seem to be about speech as expression of thought, be it opinions, theories, or art. Yvain's thoughts are about this (as are his and Vladimir's points about identifying the Right-Thinking People and legislating the truth), so the "shouting fire" example doesn't help much.

I'm not sure it's useful to speak of all three as "speech" and have a single set of general rules for them, unless this is also acting as a Schelling point (although some current debates - hate speech, blasphemy - seem to be essentially about the framing of the issue as the first or the third kind of speech, with speaker intention often acting as a [Schelling?] divide).

[edited after Lumifer's response]