Before or after extrapolation? If the former then why does that matter, if the latter then how do you know?

Former in as much as it allows inferences about the latter. I don't need to know with any particular confidence for the purposes of the point. The point was to illustrate possible (and overwhelmingly obvious) failure modes.

Hoping that CEV<humanity> is desirable rather than outright unfriendly isn't a particularly good reason to consider it. It is going to result in outcomes that are worse from the perspective of whoever is running the GAI than CEV<that person> and CEV<group more closely identified with>.

The purpose of doing CEV<humanity> at all is for signalling and cooperation (or, possibly, outright confusion).