I'm not seeing how it's going to cost you much to be seen talking to someone else, unless you're, say, clearly plotting against the rest of the village. On the contrary, popularity is kind of a self-fulfilling prophecy, in the sense that people who already have a strong social network tend to make more friends and acquaintances, thereby gaining new allies and potential mating partners. How is this a bad thing?