It's perfectly fine to be Packers fan, but I would judge a person who showed up to a wedding or funeral dressed as a cheesehead. I wouldn't judge them for being a Packards fan; I would judge them for disrespectfully violating decorum.

The second-order signal is the more negative and important. It isn't too bad if people think you are nerdy; it's bad if people think that you don't understand or don't care how most people perceive you. It signals a lack of self-awareness, or a deficient understanding of cultural norms, or blithe indifference towards non-nerds. In my case, it accurately signaled all three.

The things in bold -- I cannot put myself in the mindset where these things appear inherently bad. Would you mind explaining why should one make an end value out of respecting social and cultural norms?

Signaling my lack of self-awareness didn't cause bad things to happen; it prevented good things from happening.

I suppose you could frame it like that too. Which are those good things? (Some guesses ahead of time: getting compliments on your outfits? Socializing more easily with certain segments of society? Modifying people's perception of you into something more positive, as an end in itself or as a means to something else, e.g. a promotion?)

No negative impact, and I don't think people do assume the wrong things about me.

I assumed negative impact because I'm right on the other side of the pond: I have very good intuitions on dressing in a conventionally pretty, fashionable way, and yet this skill has always had negative value for me, as it influenced people's reactions to me in a way that worsened my interactions with them. But I'll expand on this only if it proves necessary.