Even if people have symmetric values, the relevant facts are not symmetric. For example everyone values things that money can buy, but some people have much higher abilities to earn money in a free market economy, so there will be conflict over how much market competition to allow or what kind of redistributive policies to have.

if they don't get to choose the good end of the stick

I'm not sure what you mean by this. Are you saying something like, "if they were under a Rawlsian veil of ignorance"? But we are in fact not under a Rawlsian veil of ignorance, and any conclusions we make of the form "If I were under a Rawlsian veil of ignorance, I would prefer society to be organized thus: ..." are likely to be biased by the knowledge of our actual circumstances.