For some (not all) of these cultural patterns, what happened was that the attire in question really was initially associated with thugs. In a rough neighborhood, you are either predator or prey. Even if you are not actually a thug yourself, it is safer to look like one, so that the actual predators mistake you for another predator rather than prey. It's like viceroy butterflies imitating monarchs, but without the actual evolution.
There's a lot of background mess in our mental pictures of the world. We try and be accurate on important issues, but a whole lot of the less important stuff we pick up from the media, the movies, and random impressions. And once these impressions are in our mental pictures, they just don't go away - until we find a fact that causes us to say "huh", and reassess.
Here are three facts that have caused that "huh" in me, recently, and completely rearranged minor parts of my mental map. I'm sharing them here, because that experience is a valuable one.