No need to remove it, just don't put a mic on the crowd.

My suspicion would be that the actors would appear to be reacting to something which didn't exist, and that it'd be even worse than no audience at all.

We'd have to do some experiments to check though.