Invalid logic. What the people around you generally want more of is your attention, to validate their sense of status or acceptance. If you were spending your time on some form of conspicuous consumption, this would be equally disliked as a resource drain.

TL;DR: any time you spend time and resources on something other than the people you're in relationship with, they're not going to like it that much. Altruism has fuck-all to do with it, except as your own signaling that you're a good person and the people you're with are selfish jerks.