Can you give me a circumstances-nonspecific reason?

Sure, here are a few off the top of my head, not having done any research. There are strong social norms against taking other people's property. The prevailing culture of the time held this norm. Other people owning slaves has little direct impact on most people. I don't think the economic competition argument was one likely to be known or understood by those competing with slaves, so I don't think they would have made it. People who could make that argument were busy doing other things with their wealth. Status quo bias is generally strong.

For circumstances-specific logic, I'd suggest reading up on the period debates surrounding slavery; there certainly were some at high levels. I think the decisions were mainly made on political grounds, and by people who liked the economics the way they were.