It's a refusal to participate in ritualized social bonding, and that signals you aren't willing to relax around other people and don't consider them to be part of your social in-group. If you're not drinking, that may also mean you get to keep your guard up while everyone else is saying and doing silly or even forbidden things.

I can't drink because of my medications, and I always get teased about it. "Come on, just one shot is fine..."