If you can choose to optimize yourself for interacting with the average mathematician or interacting with the average person, you should choose the mathematician.

If. That would imply that optimizing signals for "muggles" automatically sub-optimizes for "wizards".

And this is true for most values of wizard (goths, hippies, preps, nudists, professionals) but it's certainly not true when "wizards" = Intelligent, happy, kind, and effective people. You can signal allegiance to a subculture with a costume, but you can't signal a virtue.

There is no reliable superficial signal for hidden virtues and abilities which are both universally admired and difficult / impossible to acquire. If such a signal were to exist and become well known, it would immediately become fashionable and thereby lose its signaling properties.

The only honest signal for these traits is actual demonstrations of intelligence, kindness, etc...is actual displays of intellect, altruism, etc. As such, looking weirdly fancy isn't going to help you to that end. The way to surround yourself with smart/kind/effective people is to inhabit social spaces which attract smart/kind/effective people.

Smart/effective/kind people (especially the elder generations, who happen to be the most useful to impress) are often still prejudiced in all the usual ways though, so you might still alienate them with your appearance.

On the contrary: most people in the world are ignorant, useless wastes of your time.

That's a really cold way to phrase that. I think that way sometimes, but only when I'm particularly unhappy or frustrated with people. The thought goes away when I remember that most people genuinely care about other people, and many care about me (Or to put it in game theory jargon, my preferences fundamentally overlap with most people's, and I consider agents who have those preferences intrinsically valuable).