I think you should start going too. Here's something I've noticed that you may not have noticed: People reliably publicize their most awesome qualities and actions. The result is that the more superficially you know a particular person, the more uniformly awesome they seem. It's like each person has a histogram of assets ranked by awesomeness, and you only get to see a little window of that on the extreme awesome end; as the window widens, the average awesomeness may decrease. So you should imagine everyone as less uniformly awesome than they seem.

EDIT: So here's one way you could use this idea: If there's a particular person in the community who does something impressive and this makes you think "oh, I'm not as impressive as that, I don't belong in their company", you can tell yourself "well, they've probably done something really unimpressive that I haven't heard about too, and when I focus on that hypothetical failure or weakness of character, they don't seem that intimidating at all".