On a somewhat related note, non-slaveholders often bought goods that involved slave labor at some point in the process of their production. It's quite possible that they at least thought that freeing the slaves would raise the cost of their tobacco, clothes ect. edited to add This is a bit more speculative, but I suspect that labor was significaly non-fungibal, and in particular that Northerner's didn't consider Southern agricultural jobs to be closely tied to their own labor market.