Personally I find it kind of invasive when people talk to me uninvited at all. But I can hardly hold other people responsible for not complying with my own, unknown-to-them social antipathies.

Sure you could. Others manage. You could demand that no one talk uninvited to anyone else. Or demand that they only speak to you when you've spoken to them first. Or only speak to you when you want them to. Just claim that your social antipathies are the right, good, and true antipathies, and anyone violating them is a moral leper.