Elund comments on Objections to Coherent Extrapolated Volition - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (56)
Instead of thinking about a hypothetical human hunter, I find it useful to think about the CEV for dogs, or for a single dog. (Obviously the CEV for a single dog is wildly different from the CEV for all dogs, but the same types of observations emerge from either.)
I think it would be pretty straightforward to devise a dog utopia. The general features of dog values are pretty apparent and seem very simple to humans. If our technology were a bit more advanced, a study of dog brains and dog behaviors would tell us enough to design a virtual universe of dog bliss.
We are so much smarter than dogs that we could even create ways of intervening into inter-dog conflicts in ways not obvious to the dogs. We could remove the whole mentality of a dominance hierarchy from the dog's psychology and make them totally egalitarian. Since these dogs would be immortal and imbued with greater intelligence, they could be taught to enjoy more complex pleasures and possibly even to generate art.
Of course none of what I just described is actually dog CEV. It is more like what a human thinks human CEV might look like, applied to dogs in a lazy fashion. It is not Coherent in the sense that it is ad hoc, nor is it Extrapolated, in the sense that it essentially disregards what dogs actually want - in this case, to be the alpha dog, and to be perpetually gorging on raw meat.
Still - STILL - the dogs probably wouldn't complain if we imposed our humanized-CEV onto them, after the fact. At least, they wouldn't complain unless we really messed it up. There is probably an infinite space of stable, valid Utopias that human beings would willingly choose and would be perpetually content with. The idea that human CEV is or should be one single convergent future does not seem obviously true to me. Maybe CEV should be a little bit lazy on purpose, as the above human design designed a lazy but effective dog utopia.
My main point here is that this singleton-attractor phenomenon really becomes a problem when the CEV subject and the CEV algorithm become too tightly coupled. It seems to be generally assumed that CEV will be very tightly coupled to human desires. Maybe there should be a bit of wiggle room.
CEV is supposed to aim for the optimal future, not a satisficing future. My guess is that there is only one possible optimal future for any individual, unless there is a theoretical upper limit to individual utility and the FAI has sufficiently vast resources.
Also, if the terminal goals for both humans and dogs are to simply experience maximum subjective well-being for as long as possible, then their personal CEVs at least will be identical. However, since individuals are selfish, there's no reason to expect that the ideal future for one individual will, if enacted by a FAI, lead to ideal futures for the other individuals who are not being extrapolated.