This basically boils down to the root of the impulse to remove a chesterton's fence, doesn't it?

Not really. A lot of fences seem to have been taken down for bad or at least objectionable reasons, and to have turned out either fine or not to bad.

I'd point to a different distinction - effective fences tend to have more defenders than bad ones (on average). So by taking down a fence that's easy to take down, you're more likely to improve the situation. And what fences get taken down the most often? The easy ones.

So my "argument" can say that it's ok to take down a fence, but that this might not apply to major/important ones that have remained untouched to date.