Even a slight innate difference is likely to be exaggerated by other effects; in-group loyalty, halo effects, evaporative cooling, &c.

I'd file it under "sensitive dependence on initial conditions". Memetic attractors form around certain political views, and small innate differences send people spiralling to one or another.