I'm late to the party, but it's because you forgo new opportunities to meet people when you signal these things. For example, I knew a man who signalled nerdyness and had no social skills. Talking with him was uncomfortable and he would creep on women unknowingly, driving off other friends. Further, his lack of social awareness made it impossible to get rid of him once he was attached leading to an uncomfortable situation for all concerned. I had 3 other negative experiences from people who signalled nerdyness, each of which had serious problems. Essentially, by choosing to interact with new people who signal nerdyness in their dress I take on a higher than normal risk of having a negative experience.

A large proportion of people have had an experience like this and do not wish to repeat it, so by signalling nerdyness you essentially drive those people away. Aside from that, you make use of the horns effect when you could be using the halo effect. The end result is that signalling nerdyness puts a pointless obstacle in the path of your life.

Just signalling normalness puts you in a better position, because if you want to signal nerdyness to specific people you can just say "I play dwarf fortress" or something similar.