There is no safe wish smaller than an entire human morality.
Is that true? Why can't the wish point at what it wants (e.g. the wishes of particular human X) - rather than spelling it out in detail?
To spell out some of the complications -- does the genie only respond to verbal commands? What if the human is temporarily angry at someone and an internal part of their brain wishes them harm. The genie needs to know not to act on this. So it must have some kind of requirement for reflective equilibrium.
Suppose the human is duped into pursuing some unwise course of action? The genie needs to reject their new wishes. But the human should still be able to have their morality evolve over time.
So you still need a complete CV Extrapolator. But maybe that's what you had in mind be pointing at the wishes of a particular human?
r/Fitness does a weekly "Moronic Monday", a judgment-free thread where people can ask questions that they would ordinarily feel embarrassed for not knowing the answer to. I thought this seemed like a useful thing to have here - after all, the concepts discussed on LessWrong are probably at least a little harder to grasp than those of weightlifting. Plus, I have a few stupid questions of my own, so it doesn't seem unreasonable that other people might as well.