Whether we can build a “safe AGI” by giving it a carefully designed “goal system” My answer is negative. It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals.

I don't see Pei distinguishing between instrumental and ultimate goals anywhere. Whereas Luke does do this. Maybe a failure to make that distinction explains the resulting muddle.

Update - 2012-04-23 - it looks as though he does do something similar here:

In terms of source, a goal can be either original or derivative, where the former is imposed on the system, and the latter is produced from the former by the system itself. In the current design of NARS, all input tasks are original goals, and all the derived tasks are derivative goals. In human or animals, all the (biological and evolutionary) innate drives are original goals, and all the other motivations, intentions, and objectives are derivative goals.

...though original / derived doesn't seem to be quite the same idea.