My hypothesis would simply be that different people experience emotional intensity as a reaction to different things. Thus, some think we are crazy and cultish, while also totally weird for getting excited about boring and dry things like math and rationality... while some of us think that certain people who are really interested in the lives of celebrities are crazy and shallow, while also totally weird for getting excited about boring and bad things like Twilight.

This also leads each group to think that the other doesn't get similar levels of emotional intensity, because only the group's own type of "emotional intensity" is classified as valid intensity and the other group's intensity is classified as madness, if it's recognized at all. I've certainly made the mistake of assuming that other people must live boring and uninteresting lives, simply because I didn't realize that they genuinely felt very strongly about the things that I considered boring. (Obligatory link.)

(Of course, I'm not denying there being variation in the "emotional intensity" trait in general, but I haven't seen anything to suggest that the median of this trait would be considerably different in gifted and non-gifted populations.)