I had somewhat of your problem, but fortunately my comparison group consisted of famous scientists and mathematicians, and also competitive chess players -- so instead of thinking how smart I am compared to other people, I was more amazed at the stupidity I saw around me. Specific things that help put me in my place were being clobbered at chess by a kid half my height (much younger than me), getting clobbered at chess by a guy playing against 20 other people simultaneously (this happened on a regular basis with different such people), and realizing that even with the advantage of hindsight, I couldn't understand quantum and general relativity. So I've long known that I'm not memorably smart.