I fundamentally think there are very strong sacredness based feelings around this that are not based on consequences in the real world any more than other kinds of religious thinking is.

Throughout the 19th century, there have been leftist thinkers - from moderate and "respectable" ones to hardcore radicals - who either had no problem acknowledging differences in average intelligence, or were even outright racists/white supremacists. E.g., I've read that many American abolitionists either acted xenophobic towards actual black people when they met them, believed that blacks can never match whites in ability or achievement, etc. Yet their moral and religious opposition to slavery - all men are created in God's image, and ought to be treated as such - covered the immorality of one race subjugating another. So... eh, it's contradictory and messy. But ultimately egalitarianism, like all moral emotions, need not be chained to any particular empirical belief.