So we want FAI's values to be conducive with human values. But how are people proposing FAI will deal with the fact that humans have conflicting and inconsistent values, both within individual humans and between different humans? Do we just hope that our values when extrapolated and reasoned far enough are consistent enough to be feasibly satisfied? I feel like I might have missed the LW post where this kind of thing was discussed, or maybe I read it and was left unsatisfied with the reasoning.
What happens, for example, to the person who takes their religion very seriously and wants everyone to convert to their religion? Or just people whose values are heavily tied to their religious beliefs, of which there probably are a few, considering there are billions of religious people around.
There's a pdf discussing the matter linked here - though as you can see, a lot of people failed to read it (or "seer" has a lot of sockpuppets).
This thread is for asking any questions that might seem obvious, tangential, silly or what-have-you. Don't be shy, everyone has holes in their knowledge, though the fewer and the smaller we can make them, the better.
Please be respectful of other people's admitting ignorance and don't mock them for it, as they're doing a noble thing.
To any future monthly posters of SQ threads, please remember to add the "stupid_questions" tag.