Let's stop talking about race since it may or may not be relevant and deal directly with IQ.

I don't see the point of dealing with someone who is explicitly intolerant of a group of people based on no conscious choice of their own,

Someone's IQ is certainly not based on any conscious choice of their own. So your argument seems to imply that we should not be intolerant of people with low IQs.

This is because they are less likely to have interesting thoughts or experiences, and furthermore I would not feel comfortable dealing with them in many social settings.

On the other hand this argument works even better as an argument for avoiding interacting with, i.e., being intolerant of, people with low IQs.

So which is it, should we be intolerant of people with low IQs, or should we be intolerant of people who are intolerant of people with low IQs? Your argument seems to imply both.