Rich people's servants, and even middle-class people's servants, were live-in members of the household, and as such not strangers. The risks would be rather below those of an employee stealing from the workplace. Which does happen, but workplaces still work. And in a household that had someone come in to "do for them", the woman of the house would customarily have been at home.