I understand CEV. What I don't understand is why the programmers would ask the AI for humanity's CEV, rather than just their own CEV.

The only (sane) reason is for signalling - it's hard to create FAI<self> without someone else stopping you. Given a choice, however, CEV<self> is strictly superior. If you actually do want to have FAI<humanity> then FAI<self> will be equivalent to it. But if you just think you want FAI<humanity> but it turns out that, for example, FAI<humanity> gets dominated by jerks in a way you didn't expect then FAI<self> will end up better than FAI<humanity>... even from a purely altruistic perspective.

Yeah, I've wondered this for a while without getting any closer to an understanding.

It seems that everything that some human "really wants" (and therefore could potentially be included in the CEV target definition) is either something that, if I was sufficiently well-informed about it, I would want for that human (in which case my CEV, properly unpacked by a superintelligence, includes it for them) or is something that, no matter how well informed I was, I would not want for that human (in which case it's not at all clear that I ought to endorse implementing it).

If CEV-humanity makes any sense at all (which I'm not sure it does), it seems that CEV-arbitrary-subset-of-humanity makes leads to results that are just as good by the standards of anyone whose standards are worth respecting.

My working answer is therefore that it's valuable to signal the willingness to do so (so nobody feels left out), and one effective way to signal that willingness consistently and compellingly is to precommit to actually doing it.

Is this question any different from the question of why there are altruists?