Right, answering with silence seems inferior to pointing out irrelevance where appropriate (even if superior to responding on object level), it leaves the matter unsettled. So this is possibly a step that shouldn't be skipped even where irrelevance is obvious, just like with something obviously wrong. This is a natural analogy: what happens is that instead of one question, we consider two questions simultaneously: whether something is right, and whether working on figuring out whether it's right is a good idea.

For Aumann agreement, the topic is discussed on LW, so certainly isn't irrelevant.