I like this particular subject because I've been thinking about this topic as well. I do believe that the typical definitions of extrovert/introvert are a bit misleading. Dissecting what it means may be of help. For instance, when we mean energy I would translate that to mental resources. My capacity to handle stressful tasks can deplete my mental resources more. So if I were to define introvert, it would be more along the lines of being with people depletes my mental resources, and visa versa.
I don't think people are necessarily an introvert or an extrovert. It's probably easy to identify as one and feel the effects because you expect them. However, everybody is very complex and different, so categorizing people into introvert and extrovert seems to be an oversimplification. I can be outgoing or not depending on my current emotional state, the circumstances, or how mentally taxed I am. Not necessarily because I have a tendency towards introvert or extrovert. For example, when it comes to hosting a board game day, I feel lot's of "energy" because I get excited and I enjoy the activity. I'm going to want to be more social so I can increase my chances of winning. If I'm at a party, I immediately feel bored and because of my experience in parties, typically don't even want to engage with people. I also can feel mentally depleted just for being at a party. Likely, because of the stress of having to look good and having to feel like I need to make a good impression. I can guess why people would feel a burst of "energy" at parties though. I'd say it's mainly the different expectations in the setting that would drain the energy. In either setting I've both experienced a lot of mental resources and sometimes not much. That's where I think I agree with the original poster. Certain people are going to deplete your mental resources more than others, but whether you trust them or doesn't seem to make difference. I trust my sister to a large degree, but that doesn't mean she doesn't stress me out given the right situation.
Epistemic Status: This is a model I have derived from my own experience, with a fair amount of very noisy data to back it up. It may not generalize to anyone else. However, it seems like a framework that might be useful, so I'm sharing it here.
The excessively simple model of social energy is the introvert/extravert dichotomy. Introverts lose energy from social situations, extraverts gain energy. This is then elaborated into the I/E scale, where the sign of your energy change in social situations is mapped to an integer. This is clearly more descriptive of reality, but as many have pointed out, still imperfect.
I find that for me there are separate sets of factors that determine energy gain and energy loss.
For energy gain, it is a positive-slope, negative-curvature function of the number of people present. There is energy in a room, and I do pick up some of it. (Something like sqrt(n), or possibly 10-10/n
For energy loss, it is a function of how much I trust the person in the room I trust least; f(min(trust(p) for p in room)). This grows much faster than the number of people present. Trust also seems to be a function of my pre-existing mood (that part I expect won't generalize).
Naively, I would have expected this to be a weighted average of my trust of people in the room, where five people I trust very much and one I trust very little would feel very different from five I trust somewhat and one I trust very little. I have difficulty arranging that test, but preliminary data suggests that expectation was wrong; one person who I cannot relax around spoils things thoroughly. ('Trust', here, is very much a System 1 thing; feeling safe/open around someone, rather than feeling/thinking that they are trustworthy/upstanding/honest.)
The predictions made by this model are that you should choose your social gatherings carefully, even if extroverted, as the benefits of size can be wiped out by one or two poorly-chosen guests.
More broadly, I think that considering gain and loss separately will clarify the feelings toward socializing of many self-identified introverts. Since it seems quite plausible that 'true introverts' who never get energy from social interaction are not actually a thing, I expect this would help improve the day to day lives of many.