So we want FAI's values to be conducive with human values. But how are people proposing FAI will deal with the fact that humans have conflicting and inconsistent values, both within individual humans and between different humans? Do we just hope that our values when extrapolated and reasoned far enough are consistent enough to be feasibly satisfied? I feel like I might have missed the LW post where this kind of thing was discussed, or maybe I read it and was left unsatisfied with the reasoning.

What happens, for example, to the person who takes their religion very seriously and wants everyone to convert to their religion? Or just people whose values are heavily tied to their religious beliefs, of which there probably are a few, considering there are billions of religious people around.