If I decide to seek company of some people, because according to some metric M they are better than me, I am helping myself, because I am exposing myself to people better than me, but at the same time I am hurting them, because I expose them to a person that is worse than them, according to the same metric.

I am not convinced that being around people slightly worse than yourself is bad for you. Especially when you get into a mentor role. When you actively try to help others understand and improve, this forces you to think about what you are actually doing, which probably improves your behavior.

Disclaimer: purely anecdotal, and does not apply to all metrics.