A recent post at my blog may be interesting to LW. It is a high-level discussion of what precisely defined value extrapolation might look like. I mostly wrote the essay while a visitor at FHI.
The basic idea is that we can define extrapolated values by just taking an emulation of a human, putting it in a hypothetical environment with access to powerful resources, and then adopting whatever values it eventually decides on. You might want some philosophical insight before launching into such a definition, but since we are currently laboring under the threat of catastrophe, it seems that there is virtue in spending our effort on avoiding death and delegating whatever philosophical work we can to someone on a more relaxed schedule.
You wouldn't want to run an AI with the values I lay out, but at least it is pinned down precisely. We can articulate objections relatively concretely, and hopefully begin to understand/address the difficulties.
(Posted at the request of cousin_it.)
I do give a (somewhat) concise overview, in the section headed 'The Proposal.'
The 100 years example is not quite right, in that in the real example we put you in an environment with unlimited computational power. One of the first things you are likely to do is create an extremely pleasant environment for yourself to work in (another is to create a community to work alongside you, either out of emulations of yourself, emulations of others, or reconstructed from simulations of worlds like Earth), while you figure out what should be done.
That said, there are other ways that your values might change through this process. For example, one of the first things you would hypothetically realize, if you ended up in an environment with some apparently infinitely powerful computers, is that you are in a hypothetical situation. I don't know about you, but if I discovered I was in a clearly hypothetical situation, my views about the moral relevance of people in hypotheticals would change (hypothetically).
Here's another objection. Putting someone in an environment they control completely which has unlimited computational power could lead to some pretty unexpected stuff. Wireheading would be easy, and it could start innocuously: I decide I could use an attractive member of my preferred gender to keep me company and things get worse from there. If you put someone in this situation it seems like there'd be tremendous incentives to procrastinate indefinitely on solving the problem at hand.
It seems like under ideal conditions we could empirically test the behavior of this sort of exotic "utility function" and make sure it was meeting basic sanity checks.