I think there a difference between wearing clothing that just doesn't fit well and wearing the kind of weird clothing that can make a positive impression.

Wearing Hawaii shirts for example is weird but something that people who consider themselves open minded but don't consider themselves as nerds can see as cool.

I personally make the weird clothing choice of walking around in Vibriam Fivefinger shoes. In contrast to cutting myself off from other people they invite interest and people start conversation with me to ask me about them that they wouldn't start otherwise.