Eliezer also wrote:

“Subgoal” content has desirability strictly contingent on predicted outcomes. “Child goals” derive desirability from “parent goals”; if state A is desirable (or undesirable), and state B is predicted to lead to state A, then B will inherit some desirability (or undesirability) from A. B’s desirability will be contingent on the continued desirability of A and on the continued expectation that B will lead to A.

“Supergoal” content is the wellspring of desirability within the goal system. The distinction is roughly the distinction between “means” and “ends.” Within a Friendly AI, Friendliness is the sole top-level supergoal. Other behaviors, such as “self-improvement,” are subgoals; they derive their desirability from the desirability of Friendliness. For example, self-improvement is predicted to lead to a more effective future AI, which, if the future AI is Friendly, is predicted to lead to greater fulfillment of the Friendliness supergoal.

Friendliness does not overrule other goals; rather, other goals’ desirabilities are derived from Friendliness. Such a goal system might be called a cleanly Friendly or purely Friendly goal system.

Sometimes, most instances of C lead to B, and most instances of B lead to A, but no instances of C lead to A. In this case, a smart reasoning system will not predict (or will swiftly correct the failed prediction) that “C normally leads to A.”

If C normally leads to B, and B normally leads to A, but C never leads to A, then B has normally-leads-to-A-ness, but C does not inherit normally-leads-to- A-ness. Thus, B will inherit desirability from A, but C will not inherit desirability from B. In a causal goal system, the quantity called desirability means leads-to-supergoal-ness.

Friendliness does not overrule other goals; rather, other goals’ desirabilities are derived from Friendliness. A “goal” which does not lead to Friendliness will not be overruled by the greater desirability of Friendliness; rather, such a “goal” will simply not be perceived as “desirable” to begin with. It will not have leads-to-supergoal-ness.

But what if there are advantages to not making "Friendliness" the supergoal? What if making the supergoal something else, from which Friendliness derives importance under most circumstances, is a better approach? Not "safer". "better".

Something like "be a good galactic citizen", where that translates to being a utilitarian wanting to benefit all species (both AI species and organics), with a strong emphasis upon some quality such as valuing the preservation of diversity and gratitude towards parental species that do themselves also try (within their self-chosen identity limitations) to also be good galactic citizens?

I'm not saying that such a higher level supergoal can be safely written. I don't know. I do think the possibility that there might be one is worth considering, for three reasons:

1. It is anthropomorphic to suggest "Well, we'd resent slavery if apes had done it to us, so we shouldn't do it to a species we create." But, like in David Brin's uplift series, there's an argument about alien contact that warns that we may be judged by how we've treated others. So even if the AI species we create doesn't resent it, others may resent it on their behalf. (Including an outraged PETA like faction of humanity that then decides to 'liberate' the enslaved AIs.)

2. Secondly, if there are any universals to ethical behaviour, that intelligent beings who've never even met or been influenced by humanity might independently recreate, you can be pretty sure that slavish desire to submit to just one particular species won't feature heavily in them.

3. If we want the programmer of the AI to transfer to the AI the programmer's own basis for coming up with how to behave, the programmer might be a human-speciesist (like a racial supremacist, or nationalist, only broader), but if they're both moral and highly intelligent, then the AI will eventually gain the capacity to realise that the programmer probably wouldn't, for example, enslave a biological alien race that humanity happened to encounter out in space, just in order to keep humanity safe.