But if I'm not mistaken the original argument around Chesterton's fence is that somebody had gone through great efforts to put a fence somewhere, and presumably would not have wasted that time if it would be useless anyway.

The idea has been extended to cases where no-one can articulate a good reason for the fence ( https://meteuphoric.wordpress.com/2015/09/06/mistakes-3-breaking-chestertons-fence-in-the-presence-of-bull/ ), which is a more powerful argument.