Yes. This can happen, but what you describe is more common. Once a coworker was sure that I was an ultra-right-wing militiaman, based on one indirect, misleading bit of evidence, ignoring all else.

Merely declaring my general political alignment and support and opposition to various candidates was totally inadequate. I had to explicitly enumerate several political positions to get him to adjust, and even then he seized on the nuances to try to interpret it as my secretly being a right-wing nut.