I'm surprised that nobody has mentioned this one, but it's often inappropriate to hang around with people while not engaging in whatever activity they're doing. Don't sit in the stands at a football game reading a book, don't watch TV while your friends are playing Dungeons and Dragons, don't have your headphones on while friends are having a conversation.

For many people, alcohol raises talkativeness and lowers inhibition, so you're more likely to say things you normally wouldn't (in vino veritas). Sharing private things is a friendship-builder (HPMOR 7), but it can also be embarrassing. Drinking is a pre-commitment to build friendship through potentially embarrassing interactions, and when you abstain, you're saying, "I'll hear your secrets, but keep mine, thank you very much," which is a suspicious and untrustworthy kind of stance.

To the extent the above is true, it's too bad, because

1. Some people really don't like drinking, and alcohol doesn't make them more sociable anyway
2. No one should need to self-handicap in this way to trust and be trusted
3. It's a pre-commitment, limiting your options

I don't know, of course, but if my cohort regularly engages in a habit that leaves us at a physical and cognitive disadvantage, it really doesn't seem too surprising that we will also develop associated habits that prevent others who are not so disadvantaged from engaging with us.

Simple tradition, I expect. In many situations and cultures, consuming alcohol is simply the done thing, and not doing the done thing a surefire way of standing out. I'd also guess that people who drink in these situations expect everyone to know the social norms and agree with them (even if it's only an unconscious background assumption), and so they'll see not wanting to drink as wanting to stand out. And you know what that means.

I think this is mostly the impression of moral superiority people get from non-drinkers. If you excuse yourself for medical reasons, or imply you're a recovering alcoholic, people will mistrust you a lot less.

Bonding among humans involves being vulnerable. Shared food gives them the chance to poison you. Comedy allows you to signal to each other which social norms you don't take seriously. Drinking involves loss of coordination and talkativeness about deep issues one has.

Drinking is a signal of agreeableness and extraversion, which are highly valued status-enhancing traits. Not drinking when the context calls for it signals a lack of compliance with group norms - and does not have a status-enhancing spin to it that can be sold as "cool rebellion."

Personally one of the reasons I enjoy social drinking is the mutual lowering of barriers. In everyday interaction people are routinely concealing their true feelings and intentions. Consuming alcohol forces them to reveal more of their true feelings and provides a social setting where this is acceptable. I find you trust and understand people better having been drunk with them.

Consequently, someone refusing to drink is implicitly trying to conceal their true feelings to a greater degree, which makes them seem less trustworthy/friendly.