There's a far worse problem with the concept of 'utility function' as a static entity than that different generations have different preferences: The same person has very different preferences depending on his environment and neurochemistry. A heroin addict really does prefer heroin to a normal life (at least during his addiction). An ex-junkie friend of mine wistfully recalls how amazing heroin felt and how he realized he was failing out of school and slowly wasting away to death, but none of that mattered as long as there was still junk. Now, it's not hard to imagine how in a few itterations of 'maximizing changing utilities' we all end up wire-headed one way or another. I see no easy solution to this problem. If we say "The utility function is that of unaltered, non-digital humans, living today," then there will be no room for growth and change after the singularity. However, I don't see an easy way of not falling into the local maximum of wire-heading one way or another at some point... Solutions welcome.