Even if people have symmetric values, the relevant facts are not symmetric. For example everyone values things that money can buy, but some people have much higher abilities to earn money in a free market economy, so there will be conflict over how much market competition to allow or what kind of redistributive policies to have.

if they don't get to choose the good end of the stick

I'm not sure what you mean by this. Are you saying something like, "if they were under a Rawlsian veil of ignorance"? But we are in fact not under a Rawlsian veil of ignorance, and any conclusions we make of the form "If I were under a Rawlsian veil of ignorance, I would prefer society to be organized thus: ..." are likely to be biased by the knowledge of our actual circumstances.

What I was getting at is that humans have mostly symmetric values such that they should not disagree over what type of society they want to live in, if they don't get to choose the good end of the stick.

This seems wrong, except for extremely weak definitions of "mostly". People should definitely disagree about what type of society they want to live in, just a whole lot less than if they were disagreeing with something non-human.