You need to be far more certain of your beliefs to think that (1) "this F's views are so evil that she should be excluded from speaking to our community, compared to (2) "I shouldn't defer to this F's position on D."

I do agree with you that it's dangerous to say "my field dominates the study of this topic and you are not in my field so defer to the judgement of my field when looking at this topic" because some fields have not developed techniques which actually help them ascertain truth.