Even if there were some reason to think our current values were optimal for our current environment, which there is actually reason to think they are NOT, we would still have no reason to think they were optimal in a future environment.
Type error. You can evaluate the optimality of actions in an environment with respect to values. Values being optimal with respect to an environment is not a thing that makes sense. Unless you mean to refer to whether or not our values are optimal in this environment with respect to evolutionary fitness, in which case obviously they are not, but that's not very relevant to CEV.
all of which our zookeeper FAI should be able to move us (or our descendants) towards while carrying out necessary eugenics to keep our genome healthy in the absence of natural selection pressures.
An FAI can be far more direct than that. Think something more along the lines of "doing surgery to make our bodies work the way we want them to" than "eugenics".
I wonder if the FAI will be sad
Do not anthropomorphize an AI.
Type error. ... Unless you mean to refer to whether or not our values are optimal in this environment with respect to evolutionary fitness, in which case obviously they are not, but that's not very relevant to CEV.
You are right about the assumptions I made and I tend to agree it is erroneous.
Your post helps me refine my concern about CEV. It must be that I am expecting the CEV will NOT reflect MY values. In particular, I am suggesting that the CEV will be too conservative in the sense of over-valuing humanity as it currently is and therefore undervalu...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.