When reading the post, I immediately pictured a white guy in the 18th century saying something like "Of course we shouldn't give blacks rights based on the reasons you suggested, that would imply that the next thing would be giving rights to women!" (and the rejection of the latter being motivated by a false map of the world).
I see no reason why Armstrong-1 needs to like or at least accept the conclusions of Armstrong-n, assuming he is not yet aware of all the inferential steps. The premise of CEV, as I understand it, is that each step by itself is sound and in accordance with the terminal value(s) of Armstrong-1. Whether this works and whether we can check if it worked is a different question.
Even if you think changing one's stated (as opposed to terminal) utility function is always done based on comparing outcomes for intuitiveness and consistency, a position I'm sympathetic towards, it would be irrational for Armstrong-1 to reject Armstrong-n's values just because they lead to very counterintuitive conclusions. The reason being that Armstrong-1 may not be aware, due to ignorance or mistakes in reasoning, that his own values imply conclusions that are even more absurd.
In general, I think people on LW are way too quick to declare something to be their terminal value (which is of course somewhat of a self-fulfilling prophecy).
Of course the sequence presented seems ridiculous, and it were the actual output of CEV, I would be virtually certain that something went wrong. However, that is based on the reasoning outlined at each step, not due to the final conclusion. All I'm saying is that Armstrong-1 has no vote on the final output before actually having gone through all the arguments. Not even if the outcome would be something counterintuitive like negative utilitarianism. (BTW, I have yet to hear a take on population ethics that doesn't include conclusions that are highly counterintuitive!)
(BTW, I have yet to hear a take on population ethics that doesn't include conclusions that are highly counterintuitive!)
You will never hear of such a take, since it's been shown that all population theories will violate at least one highly intuitive criterion of adequacy. See Blackorby, Bossert & Donaldson (2003) and Arrhenius (2000). Unfortunately few people in this community seem to be aware of these results.
My main objection to Coherent Extrapolated Volition (CEV) is the "Extrapolated" part. I don't see any reason to trust the extrapolated volition of humanity - but this isn't just for self centred reasons. I don't see any reason to trust my own extrapolated volition. I think it's perfectly possible that my extrapolated volition would follow some scenario like this:
There are many other ways this could go, maybe ending up as a negative utilitarian or completely indifferent, but that's enough to give the flavour. You might trust the person you want to be, to do the right things. But you can't trust them to want to be the right person - especially several levels in (compare with the argument in this post, and my very old chaining god idea). I'm not claiming that such a value drift is inevitable, just that it's possible - and so I'd want my initial values to dominate when there is a large conflict.
Nor do I give Armstrong 7's values any credit for having originated from mine. Under torture, I'm pretty sure I could be made to accept any system of values whatsoever; there are other ways that would provably alter my values, so I don't see any reason to privilege Armstrong 7's values in this way.
"But," says the objecting strawman, "this is completely different! Armstrong 7's values are the ones that you would reach by following the path you would want to follow anyway! That's where you would get to, if you started out wanting to be more altruistic, had control over you own motivational structure, and grew and learnt and knew more!"
"Thanks for pointing that out," I respond, "now that I know where that ends up, I must make sure to change the path I would want to follow! I'm not sure whether I shouldn't be more altruistic, or avoid touching my motivational structure, or not want to grow or learn or know more. Those all sound pretty good, but if they end up at Armstrong 7, something's going to have to give."