The steelman opponent of the fence is "I see a reason not to have the fence, and any benefits to having the fence is outweighed by the benefits of removing the fence."

By contrast, the Chesterton's Fence argument is that there are unrecognized benefits of the fence. In practice, this easily devolves into an argument about the relative costs and benefits, but that is probably a distinct argument.