For what it's worth, you've echoed my experience precisely.

It's easy and good to talk in one-on-one or small groups of people. It's also easy for me to speak publicly or in front of an audience. But interaction in strictly social settings - parties, bars, clubs - is almost painful. I don't know how to begin. The impression is just that of trying to start climbing a wall that admits no footholds.

I'm pretty certain I don't fall very far on the Asperger's spectrum. In particular, I can read body language and subtext pretty well, and I'm considerate of others to a degree I haven't seen in people I know who actually have Asperger's. I suspect that, as others have said, we're conflating distinct causes under a single banner.