The political identity is not just a label you use for yourself; it usually also comes with some model of the world. It's often confused, some people using the same label may use different models, but I think that to some degree the political labels do pay rent in anticipated experiences. (People just don't check those experiences, or filter them heavily.)

For example, if I don't use any label for myself... but from my opinions it is obvious that I expect government to introduce bureaucracy and pervert good ideas, but I expect private companies to come with good ideas if given proper financial incentives... and someone else expects government to act in the interest of powerless opressed masses, and expects rich and powerful individuals to destroy everything for their own profit... then we kind of didn't use labels, but it's not like we can't easily infer them.

Perhaps even this is a step towards sanity. At least we make our models more visible, and are more likely to come with specific predictions, which may be falsifiable. But we can still be in a situation where our models recommend completely different actions, and an experiment to decide would be too expensive (e.g. we would have to work for 5 years using one model, and at the end we would have some evidence whether the model was correct). Although this situation might happen with some issues and not happen with other issues, so even some issues could be solved together.

A perfect outcome would be, if we could someone all have an approximately correct model of the world, and help each other propagate our beliefs. But I guess at that moment we would already stop using the political labels naturally. (And we would be already assigned some label(s) by the rest of the society. Probably just rounding us to some nearest cliche.)