First, the bar for "guest speaker" is lower than for "tenured faculty." Yes, importance comes in degrees.
Ok, what about Kathy Boudin or Bill Ayers?
I have a problem with privileging bad ideas
The question is who determine which ideas are bad.
A high burden of proof for both.
The answer is that about everybody makes this determination whether you want them to or not.
Related: Heuristics for Evaluating the Soundness of the Academic Mainstream, Admitting to Bias, The Ideological Turing Test