Is this just because we have no practice talking to people of average-ish intelligence?

We tend to get used to (read: lazy in a system 1 time-saving way.) the systems we are regularly engaged in. i.e. I don't often talk about the weather (and other small talk) with people of higher intelligence (unless in a meta-sense). So I sometimes feel off-guard when trying to respond in the "normal" way that people do.

Is it because intelligence gaps are inherently difficult in social settings?

I usually appeal to stereotypical behaviour where there is common ground. For example - "growth mindset vs fixed mindset", The last time I talked to someone in the fashion industry I equally had a "black box" discussion, and found the bad communication experience too. But then I went on to talk about how I don't know because I had never ever gotten into it, and asked them to "teach me everything you know", which is a way to stump them back in the other direction. At the very least it get's people thinking, at the most you might get the opportunity to learn, they might get the opportunity to think, and you establish a new connection with someone you didn't think was interesting.

Is there some factor other than intelligence that's causing this? Are we just socially inept?

There is something called "social intelligence" which you might not be optimising for. Which is completely fine, but can cause this sort of experience to happen. I have personally recently decided to invest more in social-intelligence. The main take-away from that concept is that there are only so many hours in the day. If you want to expend 100 hours on social intelligence learning, that's 100 less hours on learning programming or tackling UFAI, Or learning about music, fashion, communication, art, biotechnology, medicine, law... There are many things that people find valuable. Not everything is universal. I would consider social intelligence as a meta-skill, in a similar way to improving your memory, increasing your sleep quality, learning to communicate in another language, and 101 other things.

It's up to you whether you want to pursue social intelligence.