Is this just because we have no practice talking to people of average-ish intelligence?

I believe a lot of practice would make talking to people with average intelligence easier. Not necessarily more satisfying, though. You will always have to debate with them at their level, while you probably enjoy debating at your level.

Is it because intelligence gaps are inherently difficult in social settings?

Could be. We naturally model other people by imagining what we would do in the same situation. If you use this algorithm to model people with much lower intelligence, you will get many predictions wrong.

Is there some factor other than intelligence that's causing this?

Sure. Rationality, culture, past experiences. All of these make modelling each other more difficult.

Are we just socially inept?

Possibly; but what exactly do you mean by "inept"? Unable to learn? -- probably not. Haven't learned yet? -- probably yes. The question is whether learning "how to communicate with normal people" is the best use of your time.