Critical discussion is important, but there's a time and a place. In the speaker-audience model, I'm not sure we should expect the speaker to present much evidence for the things that they say. The format isn't well-suited to it, when neither the speaker nor the audience is capable of looking up references. It would be better for the speaker to prepare a list of references and share it outside the talk, and for the talk itself to focus on the things they're actually trying to say, which perhaps the speaker-audience model has a comparative advantage for.

I do agree that it might not be obvious whether a question is valuable, so perhaps a better rule of thumb would be "if the speaker answers a question, don't follow up on the reply".