Oops.

Yup, I missed something there.

Well, for humans, we've evolved desires that work interpersonally (fairness, desires for others' happiness etc,). I think that an AI, which had our values written in, would have no problem figuring out what's best for us. It would say 'well, there's is complex set of values, that sum up to everyone being treated well (or something), and so each party involved should be treated well.'

You're right though, I hadn't made clear idea about how this bit worked. Maybe this helps?