People get rich by owning capital, and it's hard to compete with them if you don't. What's the difference?

Workers compete with other workers, not capitalists. Worker wages are positively associated with the level of capital accumulation in society.

In the slavery example, free workers compete with slaves, not slave owners.