For FAI to be possible, CEV() must output for (A) any morality function or (B) set of morality functions
Any set? Why not just require that CEV.HUMANITY() be possible? It seems like there are some sets of morality functions G that would be impossible (G={x, ~x}?). Human value is really complex so it's a difficult thing to a)model it and b) prove the model. Obviously I don't know how to do that; no one does yet. If moral realism were true and morality were simple and knowable I suppose that would make the job a lot easier... but that doesn't seem like a world that is still possible. Conversely, morality could be both real and unknowable and impossibly complicated and then we'd be even in worse shape because learning about human values wouldn't even tell us how to do Friendly AI! Maybe if you gave me some idea of what your alternative to anti-realism would look like I could answer better. In short: Friendliness is really hard, part of the reason it seems so hard to me might have to do with my moral anti-realism but I have trouble imagining plausible realist worlds where things are easier.
First, a terminology point: CEV.HUMANITYCURRENTLYALIVE() != CEV.ALLHUMANITYEVER(). For the anti-realist, CEV.HUMANITYCURRENTLYALIVE() is massively more plausible, and CEV.LONDON() is more plausible than that - but my sense is that this sentence depends on the anti-realist accepting of some flavor of moral relativism.
Second, it seems likely that fairly large groups (i.e. the population of London) already have some {P, ~P}. That's one reason to think making CEV() is really hard.
...Human value is really complex so it's a difficult thing to a)model it and b) pr
From the last thread:
Meta: