Maybe part of the point of this question was, much like the injunction to write our own commandments, an exercise in determining what we would do in a God-like position- such as we might be if we were to design the first successful fAI and tell it what it should want to do.