The same psychological processes that make you not want to see strangers in your house might also make you uneasy about having strangers in your house when you aren't even there to see what they're doing. (Anecdata: I would feel a bit creeped-out about either, and a bit kinda-rationally worried that they might steal things etc. Both worries would be worse in the case where the stranger is in the house and I'm not.)