If that is the case, then it is applicable even less often than it is applied, since it tends to be applied in cases where the removers have reasons that go far beyond "not knowing it is there". Invariably, they want something removed because they don't like its consequences.
Your statement doesn't look falsifiable.
Chesterton's meta-fence: "in our current system (democratic market economies with large governments) the common practice of taking down Chesterton fences is a process which seems well established and has a decent track record, and should not be unduly interfered with (unless you fully understand it)".