Friendliness researchers also need to study what human values actually are and how they are implemented in the brain.

There is apparently a pervasive assumption (not quite spelled out) that a general theory of reflective ethical idealization will be found, and also a general method of inferring the state-machine structure of human cognition, and then Friendliness will be obtained by applying the latter method to human cognitive and neuroscientific data, and then using the general theory to extrapolate a human-relative ideal decision theory from the relevant aspects of the inferred state machine.

I think this is somewhat utopian, and the efficient path forward will involve close engagement with the details of moral cognition (and other forms of decision-making cognition) as ascertained by human psychologists and neuroscientists. The fallible, evolving "first draft" of human state-machine architecture that they produce should offer profound guidance for anyone trying to devise rigorous computational-epistemic methods for going from raw neuro-cognitive data, to state-machine model of the generic human brain, to idealized value system suitable for implementing friendly AI. (For that matter, the whole process also needs to consider the environmental, social, and cultural embedding of human cognition - the brains whose volitions we want to extrapolate are human brains that grow up in humane supportive societies, not feral wolf-child Robinson-Crusoe brains that never learn language or socialization.)

So the ideal curriculum needs to contain an element, not just of formal decision theory, but of the empirical study of human decision-making. But I'm not sure where that is best covered.