People with strong political identities usually have their maps systematically distorted.

Oh, certainly. Feminism points out, though, that the social mainstream is also a strong political identity which systematically distorts people's maps. They use somewhat unfortunate historical words for this effect, like "patriarchy". That's just a label on their maps, though; calling a stream a creek doesn't change the water.

So combining this with your guideline, we should be careful not to invite anyone who has a strong political identity ... but we cannot do that, because "ordinary guy" (and "normal woman") is a strong political identity too. It's just a strong political identity one of whose tenets is that it is not a strong political identity.

We don't have the freedom to set out with an undistorted map, nor of having a perfect guide as to whose maps are more distorted. Being wrong doesn't feel like being wrong. A false belief doesn't feel like a false belief. If you start with ignorance priors and have a different life, you do not end up with the same posteriors. And as a consequence, meeting someone who has different data from you can feel like meeting someone who is just plain wrong about a lot of things!

Also ... I wonder what a person whose maps of the social world were really "no better than random" would look like. I think he or she would be vastly more unfortunate than a paranoid schizophrenic. He or she would certainly be grossly unable to function in society, lacking any ability to model or predict other people. As a result, he or she would probably have no friends, job, or political allies. Lacking the ability to work with other people at all, he or she would certainly not look like a member of any political movement.

As such, I have to consider that when applied to someone who clearly does not have these attributes, that expression is being used as merely a crude insult, akin to calling someone a "drooling moron" or "mental incompetent" because they disagree with you.