discharge them without a reference... with no welfare state they faced destitution.

Isn't that the same as kicking them out on the street without a social net?

In modern times, you are right about the use of illegal aliens, who are in a weak position to resist exploitation.

That's true, but my point was more that being vulnerable to exploitation also leaves them vulnerable to retaliation for their actual misdeeds. You might be willing to pay your servant well, but the same factors that ensure you don't have to pay your servant well also let you prevent your servant from stealing and breaking things.